The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for RL | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

In this video I cover Google Brain's "The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning" paper. They introduce the "Attention Neuron" model that copes with arbitrary permutations of input observations.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2109.02869
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 A high-level overview, main ideas
08:00 Permutation invariance vs equivariance
13:00 PI implemented via the Set Transformer idea
19:20 Interactive demos (blog)
22:00 Results
26:15 Pong occlusion experiment
28:00 Representations visualized via t-SNE
29:30 Attention Neuron is robust, attention visualized
31:45 Outro, recap

Credits:
SmarterEveryDay video: https://www.youtube.com/watch?v=MFzDaBzBlL0
Cellular Automata video: https://www.youtube.com/watch?v=C2vgICfQawE

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#attentionneuron #permutationinvariance #googlebrain
 What's cracking guys? In this video I'm covering this new paper called the sensory neuron as a transformer permutation invariant neural networks for reinforcement learning by Jujing Tang and David Ha from Google Brain. So the idea is to create an agent that will be able or a framework that will be able to solve a problem like this one. So you have a RL like environment here like this car racing game and the goal is to be able to to play the game even though the observation was severely shuffled. So we had some certain type of permutation being applied to the input observation. So as you can see here this is our input, this is your the original car racing game and then you divide this like observation into patches like a grid of patches and then you do arbitrary permutation and you get something like this and you still want to preserve the the performance that the agent had previously. So obviously if we took something like the DQN agent that's one of the most famous and one of the first successful like RL agents that were able to play these types of games, if you know how it works you know that it's going to fail miserably here because if you learn how to play on this observation it's using CNN in the background remember and so it's going to completely crash here because the like initial structure the original structure the spatial structure that this observation had is not is no longer present here so it's going to fail. If you haven't watched my DQN video do check it out I'm going to link it somewhere here. So how they managed to achieve this is there are two components to this. So the first component is this they borrow ideas from this self-organization literature and the second idea is to add the permutation invariance like to bake it in into the model and that way they achieve that this this behavior I just explained. So let's see what I see here. Numerous studies have demonstrated that humans can adapt to changes in sensory inputs even when they are fed into the wrong channels but difficult adaptations such as learning to see by interpreting visual information emitted from a grid of electrodes placed on one's tongue or learning to ride a backward bicycle require months of training to achieve mastery. So I mean just the pure fact that humans are able to do this is fairly mind-blowing. So this first thing learning to see through the tongue is you basically have like a camera I guess and that camera approximately contains the same information as what your eye sensor would receive and so you just take the electrode so the information coming from that RGB camera you place it on someone's tongue and the like neural plasticity of the brain does the rest after obviously long like multiple months of training but you can learn how your brain can learn and your neurons can adapt like to form a circuit that can understand the image even though it's coming from a different channel so that's like tongue so that's something called sensory substitution. Like check out this video from getting smarter every day where you have this backward bike and the goal that the idea is that the input observation is kind of permuted that means that you try to turn the bar left but the wheel turns right so you have this discrepancy this this shuffling that happened and all of a sudden nobody can actually ride that bike even though you know how to to ride the normal bike and so I guess there is an analogy between that and this thing here it's just that this is a bit more complex and you can imagine we couldn't be able to play this without a severe like a lot of training. So check out this short clip. Where I work the welders are geniuses and they like to play jokes on the engineers. He had a challenge for me. He had built a special bicycle and he wanted me to try to ride it. He had only changed one thing when he turned the handlebar to the left the wheel goes to the right when you turn it to the right the wheel goes to the left. I thought this would be easy so I hopped on the bike ready to demonstrate how quickly I could conquer this. And here he is ladies and gentlemen Mr. Justin Sandlin first attempt riding the bicycle. All right so the faster I go the better. Yeah yeah I couldn't do it. You can see that I'm laughing but I'm actually really frustrated. In this moment I had a really deep revelation. My thinking was in a rut. This bike revealed a very deep truth to me. I had the knowledge of how to operate the bike but I did not have the understanding. Okay so now the question is can we do better and create AI systems that can rapidly adapt to sensory substitutions without the need to be retrained. Can we avoid the need to retrain the network in order to complete this task. So even humans need months to adapt to this novel task. So this paper is in a way trying to go even above the human intelligence at least in this very particular domain. One more interesting idea I want to like read out here and then I'm gonna dig into the paper is this. In complex systems we often observe complex global behavior emerge from a collection of agents interacting with each other in their environment with each individual agent acting only on locally available information without knowing the full picture. So there are systems such as they mentioned here an implementation of these systems such as cellular automata where basically all of the cells all the components of the system only have access to a local information and then they are able to somehow communicate and form this global representation this global complex behavior. If you don't know what cellular automata is check out this short video. So basically every single cell consults the neighbors and uses their information to update its own information and every single cell does that for itself and then you have this global like behavior emerging which is pretty amazing. So a similar idea was applied in this paper basically every single one of these patches will be fed into a unique neural network all of them will be identical the ways will be shared but like still we have like a single neural network will have access to only like a local information so for to a single patch and we somehow need to integrate that information into a global representation which can then be fed into some policy network of an RL agent and then the agent can learn how to play this thing. So the second important component you want to bake in into this model is that no matter the permutation of the patches in the case of these visual games you want to have the same global representation so that means we need to have some kind of permutation invariance baked in. The way this paper achieves it is using some the idea from the set transformer paper. Now let me let me break it down and explain all of that. Okay so here it is. So the set transformer cleverly replaced Q and again I'm gonna assume you know what QKV is it's a like a terminology from the original transformer paper if you're not familiar with it just check out my transformer video. So they replaced the Q with a set of learnable seed vectors so it's no longer a function of input X thus enabling the output to become PI or permutation invariant. So before I dig into this chart here and explain how exactly did they accomplish like the permutation invariance let me quickly explain the difference between permutation equivariance and permutation invariance. So they mentioned here using these like formulas basically this property needs to be abated if you want to have this is by definition what permutation invariance is all about. So you have some input X you do some permutation over that input X so that's you can treat it as a list of tokens and then once you index it with this S S is just a permutation of this list here and you basically index X which means which is a maybe a non-Pi notation I don't know if this is like I've never seen this notation in math notebooks but basically what you do here is you're permuting the input X according to this like list S and once you pass that through function F you want to make sure that it's the same as the original like as passing the original input. So no matter the permutation you apply to the input you get the same output that's the permutation invariance and on the other hand we have and as you can see here the interesting thing here is that the input space dimensionality does not have to be the same as the output space dimensionality. On the other hand if we have permutation actually variance the those two have to match so we we gotta have and both in the input space as well as in the output space. So here is the formula for for permutation actually variance as you can see here so now once we permute the input what happens is because we have the equivalent part basically the output is going to permute in the same manner. So the original transformer paper had exactly this property and all of the transformers except for the set transformer have this property. So basically let me try and explain it really quickly. So let's say we have three input tokens so a red token we have a green token and we have a blue token. So we pass them into the transformer layer so we have like a black box here this is going to do the transformer magic and out comes some novel representations. So I'll just gonna denote them as this double bar red and then so let me take the green one double bar green so that's some novel representation and finally the blue one. So the the reason why transformers are permutation equivalent is because if we imagine I'm just gonna permute this input now like this. So we're gonna have red here we're gonna have green here and we're gonna have blue remain on the same position. So what is now going to happen is that we're just going to permute the output the same way as we permuted the input. So that's what this formula here is telling us. Basically that's what transformers are going to do by default. So we're gonna have the following thing we're gonna have green will be here red will be here and blue will be here. Okay so let me make the permutation explicit. So we have we have this is the original like ordering of the inputs one two three what we did here is we put two here because two corresponds to green one then we had one then we had three. So this is the permutation we applied to the input. So now what this formula tells us is that passing this permuted input through H and this is H in our case so I just you know the T, T's are H. So passing this in the input so this novel permutation so that means this thing here should be the same as passing as passing the original input so that means this one here passing it as the so the output of that thing is going to be this thing and then we apply the the permutation so that means we permute this thing like this. So we're going to take the the second output which is this and put it in the first place we're gonna take the first output and put it in the second place we're gonna take the third output and just leave it where it is and as you can see we get exactly this output here. So hopefully this was clear maybe a bit confusing but yeah this is the this is the permutation equivariance. So now what we're trying to accomplish is instead of having this the outputs being permuted the same way as the inputs we want to have them remain fixed. So that means for example we want to have this type of arrangement no matter how we permute the input. In this case we'll have like three factorial type like possibilities to permute the input and we'll always have this output so that will be the permutation invariance. Now one more important detail between permutation equivariance and invariance is that in the case of equivariance the like the number of input tokens and I'm going to use the terminology from transformers has to be the same as the number of output tokens. Whereas once you have the permutation invariance you can have for example arbitrary number of tokens here like n and you can still keep this maybe I don't know like five whatever. So that's another important detail and we'll soon see why that is important basically we can some of the patches of the input patches we can kind of omit them and the agent will still be able to play the game. Nice now let me try to explain how the set transformer idea comes along here. So this is the input token these are the input tokens so we have in the case of the image all of these will be just like a single patch so this ot means observation at time t and we are taking patch number one and we are feeding them into these sensory neurons. So as I said these are going to be shared so that's the idea borrowed from the self-organization literature you basically have like this network weights are going to be shared across every single one of sensory neurons and basically as you can see every sensory neuron will have as an input just the local information so just in the case of images just a couple of pixels in that single patch. So what will happen is all of them are going to generate the keys and the value vectors so that's your usual transformer thing and then instead of generating query vectors as well we decouple that part we decouple this part as you can see from the input so that means that this Q matrix is just going to be like a fixed embedding like a like an embedding table you can think of it this way like imagine we have just three vectors here okay and that means that if you know how transformers work that means that we're always going to have three vectors at the output here so this message will have constant format of three vectors like this no matter the number of inputs so no matter the number of patches we input we're always going to have three vectors as the output because that's dictated by the Q matrix. Let me now try and explain why this thing is permutation invariant so if we take this particular query vector this one here and we do a dot product with all of the keys here so we'll end up with with coefficients alpha 1 alpha 2 all the way through alpha n okay and we're going to have value vectors basically v1 v2 all the way through vn and as you remember as recall we just do multiplication between the attention coefficients and the value vectors we add them up and we form the message here so because of the addition that's that's the part that enables us the permutation invariance so now imagine I just permute I randomly permute the input observation so what will happen is that the key vectors will be randomly permuted as well as the value vector so maybe so then after we apply this query after we do the dot product between query and those new keys and new value vectors I mean the new order of the keys and values we'll just have this this the following thing so basically let's imagine we have something like this alpha 2 alpha 1 all the way through alpha n and because the value vectors are going to be permuted the same in the same manner we'll just have v2 v1 all the way through vn and now again because we're just doing multiplication between these and just adding them up we're going to end up with the same result and that's why we have permutation invariance so basically by doing this we bake in by design we bake in this permutation invariance property into our neural network so we don't have you do not have to learn how to handle different permutations of the inputs okay and basically on a high level what will happen with all of these agents and they tested these agents on four different games is we're going to input the observation we're going to input the previous action we're going to pass them through this attention neuron so this is this whole pipeline we just saw here we're going to form the message we're going to have some policy function here and we're going to output actions the next action that will that we will use to interact with the environment okay having explained this like how this thing works I'm just going to make a small remark here so these here are sensory neurons and the whole thing is called attention neuron so I'm just going to go back to the title the title name was the sensory neuron as a transformer and I think this part should be attention neuron I guess because that's where the transformer like attention happens and not on the sensory neuron level but yeah and anyways a super small net I may be off here let's get back to the paper okay so quick notice here so everything I just explained visually is just captured here algebraically basically what will happen is every one of these sensory neurons is gonna from the observation is going to create its own key is going to create its own value and as you can see here the inputs are a bit different the key function will be accepting the observation so this like the ith component as well as the previous action whereas the value like functions will only be accepting the observations and now I guess the reason for that is you do not want to bias the like current action with the previous one because remember how this thing works is you're gonna just have in the this message which is just going to contain like a sum a weighted sum of value vectors and so these value vectors will not have like the previous section like incorporated into them and so that means we won't won't be biasing the the current action at this time step so I may be completely wrong here but I think that's that's the reason they omitted the last action for the value function here finally this is just the transformer formalism the important thing to notice here is that the q does not have any input and that's the way we achieve the permutation invariance and the second thing is they decoupled like the q matrices with this wq's which is not usually done because once you apply the wq's you actually get the queries and the keys and the values so here they just decouple this to to make this formalism a bit more clear I guess but again if you understood the what I explained here you don't need to worry about the formulas okay so a couple more things they have four experiments they have these non vision continuous control tasks like the this this like card pull swing up and they also have vision based tasks like the pong game and the car race game and I'm gonna switch to blog and show you the animations and then I'm gonna return back to the paper and show you the the results they got okay here we are basically we have a like this card pull problem it's a very famous toy problem in the RL field you're trying to just basically make sure that this card is at the x equals zero position along this x-axis and you're trying to make sure that the theta angle equals zero that means that this bar the pole is vertical and so what will now happen as you can see here these are the observations so we have five we'll have five sensory neurons well one will be processing at this point like this theta dot which is angular velocity then we have cosine and sine of theta we have the x velocity and we have the x position because of the way this agent is implemented and that's using LSTM for the sensory neurons that means once I click shuffle observation it will take some time for the hidden state of the LSTM to recover and understand which input is parsing and that's what we'll have a small transition once I hit this and then we'll the agent will still manage to to achieve the performance and to balance the pole so let me let me click shuffle observations okay so nothing happened there was just some small glitch there let me touch it again so this time it failed but it's gonna manage to balance it out again so that's one example the non-visual task they were they were using their experiments there is this is the second experiment they've done and that's using the pong game and as you can see here aside from shuffling they also mask certain patches like certain patches out and you can imagine this would be super hard for a human to play and the agent actually manages to to to play it really well masking out the the patches is possible because of the m versus n so the input and the output space do not have to be the same that means we can change the number of like components of the observation at the input and we're still going to have the same message at the output and the policy will be able to handle it so that's why we can handle these like patches so yeah aside from that we have like the car race game and they showed that they can that their agent can actually handle different backgrounds even though the agent never saw these different backgrounds during the training and the reason it can achieve that is because the representation that this like model like constructs is basically abstracts out the background as a not important information to play this game but we'll get back to to those details a bit later this is just for you to understand and have a clear picture of the tasks they were using in this in in their experiments now quickly getting back to the paper let's digest the results they got so this is the carpool like a static image and again kudos to the authors for making this interactive demos in their blog i think that's the the way forward because pdfs are such an obsolete format in my honest opinion but yeah um okay so let's see the results they achieve they have two baselines first is fully connected neural network with five observations the second one is fully connected neural network with 10 observations so these fives are recalled that we had like the x the uh the velocity we had the theta the angular velocity etc so those are the five observations these baselines will be just you just input the five observations into a fully connected layer you get a intermediate hidden representation you you pass it to another fully connected layer and out comes the output so these are these baselines and the this ours is just instead of using here a feed forward network we're going to use this attention neuron we just developed so this is going to be attention neuron component here attention neuron okay so these are the models we are comparing with and as you can see here on the five observations it's a bit it's lagging a little bit behind the baseline and obviously we have uh like no answer here because we cannot plug in five observations into this neural network which is expecting 10 observations so that's the uh the another con of these baselines they are not flexible they are not adaptable to the variation of the number of inputs so the reason there is a small lag here is because they're using lstm so it takes some time to to build up to understand the input that the that particular sensory neuron is is parsing and uh now makes me wonder whether we can just use uh instead of lstm's just use the fully connected network and achieve better results here i'm not sure i saw in the paper like some ablation like that one and i guess it should work since this thing is working already i don't see why this wouldn't work but yeah anyways uh the the the point here is that once we shuffle the observations uh this thing completely fails and the attention neuron a model uh doesn't have any problems dealing with shuffled uh like inputs then uh expanding to 10 observations which is proving the point that this model can handle varying number of inputs you can see that result is still like holds here here we don't have an answer because this is as i said five observations and we we can only test the 10 observation uh fully connected neural network and again it's lagging a little bit behind it uh finally the interesting column for me is this one because um instead of adding uh and instead of duplicating the the five uh input observations this time they actually uh add random gaussian noise and the the query vector somehow learned to ignore that information and only focus on the actual uh signals so basically if i go if i scroll back to this chart uh that means that these query vectors are learned such that even though we have even though that like a half of these observations are like pure noise it will learn how to ignore these and so i guess the the the the corresponding attention coefficients will be super small and so the value vectors will be smaller that were emitted from these observations and so the message will be constructed only from the mostly from from the useful signal uh parts of the observation and not the noise and that's fairly fairly intriguing property that that happened there on this diagram they just visually present how the message so the the uh global representation is the same so basically in the case of a card pool uh this empty vector so let me let me scroll back here so this empty is just a 16 dimensional uh like a vector and they just plot it here so this here you pick one time step and this will be the empty at that time step and as you can see here these two plots are the same which is nice because uh here we don't have the observation shuffling and here we have the observation shuffling and the message remains the same which means this is just a visual representation of the of the of the permutation invariance property uh okay uh so i already showed you uh the demos and there is just they just did a bit more um they were very thorough with these experiments so what they did is they did this uh training occlusion ratio they they they changed it from zero to 0.9 which means zero means you you don't have any occlusion so all of the the patches are actually uh contained contain the signal they are not like blacked out and 0.9 means 90% of these patches will be blacked out and so training the agent across that range and then testing it again on various ratios of occlusion you can see an interesting pattern emerging here so we definitely see the best results for 50% and so as a quick reminder for the game of punk uh 21 is the top score you can achieve so you're that you have two paddles here you're playing the game and you can uh once the the ball gets past your opponent that's plus one for you and you can get until 21 so because these are averages um 21 means you're pretty much constantly winning the game and so as you can see here until we get to approximately 50% of occlusion in the test we are performing really well uh and then it drops it seems that the agents are struggling if we have too much occlusion during training but this this one this one is they mentioned this example here and it's interesting because when you when you basically train with 80% occlusion and then you show all of the patches in the test time you can see that it's still on pair with the opponent meaning that on average it's even a bit better than the opponent when you have all of the patches which means it can leverage the additional information and generalize to it even though it did not have it during the training time okay they also analyze the representations they get from this attention neuron so we just what they do is they take the empty vector they just apply t-sne and they just project the empty to two-dimensional space so if we take uh like outputs which are close to each other so empty is projected nearby in the 2d space we can see that the corresponding inputs that led to that empty being projected here are fairly similar so here you can see that this uh i guess this is called pad or paddle uh is all the way down and in all of these inputs so basically this is a single input this whole row is a single input because again it would be helpful if you knew how dqm works especially when you're dealing with these Atari games what you do you stack four consecutive frames to avoid having this uh perceptual aliasing whereby you do not know if you see a single image whether the ball is going in this direction or that direction or that direction so you do not know the state once you concatenate couple frames you have uh like a less uh like a vagueness let's go that way in your input so you can see that the inputs are are visually and semantically similar and they are grouped here into the same uh like part of this 2d space here you can imagine once you have this type of representation the agent can learn how to to make good decisions and play play play the game of punk okay um i mentioned this showed you i showed you this in the blog uh basically uh the cool thing is that the uh like our this model the attention neuron manages to to cope with different types of backgrounds so they they tried four different backgrounds here uh plus the shuffling and they showed that the like agents still can't play the game because it's ignoring uh that irrelevant information which is the background in this case and they plotted on this diagram here the attention so how they construct these representations is remember these are patches this is going to be your input it you're going to pass it to the sensory neuron and it's going to emit the key and so once you you take your query you just do dot product with all the keys and you can see that these here uh get the most attention which is basically the like the the curve the turning so i mean you as humans we can we can notice that these are very important because once you take the features that correspond to these patches and you combine them into the message and then you can basically use that feature that novel global representation to make a good decision and that's to turn left in this case similarly here even though they swapped the background so the original background is this screen like a grass type of thing once you swap it with this random background the the model still learns how to uh pay attention to the relevant parts of the image and so it knows when to turn i think that's pretty much it uh there is one more interesting thing here and that's uh they not only shuffle the input observation they do that dynamically during the the game so every for example t equals 25 steps they shuffle the input observation and they do that from 25 50 all the way to 500 and no reshuffle and we can see the results here are pretty pretty nice so looking at the car racing uh the the performance drops somewhat same as for a carpool but for different reasons so here we have the lstm so it takes some time to update the hidden state here they have some different strategy they're using some difference between the the the neighboring frames and so once you do the uh like a shuffle that means that that difference is going to have a it's going to introduce an error into the model and that's it's going to struggle uh like uh until it stabilizes again so there are a bunch of details they did a very nice and thorough experimentations across different games you can check out the blog yourself and the interactive demos but basically the main ideas are uh to have this type of sharing uh between sensory neurons and to add the the set transformer idea so that we have the permutation invariance and so that we can shuffle the observation and still preserve the same empty upon which we are building the policy network so i haven't been focusing obviously on the ariel component because that's just um not the the core inside behind this paper they were also using evolutionary evolutionary strategy algorithms to basically train the agents but the main idea was to test whether the agents can be more robust to permutation to random shuffling of the input uh observations so hopefully you found this video useful if you did share it out with a friend subscribe this channel join the discord community there is a lot of cool conversations happening there people are helping each other out so do join it and until next time bye bye
[{"start": 0.0, "end": 3.2800000000000002, "text": " What's cracking guys? In this video I'm covering this new paper called"}, {"start": 3.2800000000000002, "end": 8.32, "text": " the sensory neuron as a transformer permutation invariant neural networks for reinforcement"}, {"start": 8.32, "end": 15.36, "text": " learning by Jujing Tang and David Ha from Google Brain. So the idea is to create an agent"}, {"start": 15.36, "end": 20.64, "text": " that will be able or a framework that will be able to solve a problem like this one. So you have a"}, {"start": 20.64, "end": 28.64, "text": " RL like environment here like this car racing game and the goal is to be able to to play the game"}, {"start": 28.64, "end": 34.4, "text": " even though the observation was severely shuffled. So we had some certain type of permutation being"}, {"start": 34.4, "end": 40.32, "text": " applied to the input observation. So as you can see here this is our input, this is your the original"}, {"start": 40.32, "end": 46.96, "text": " car racing game and then you divide this like observation into patches like a grid of patches"}, {"start": 46.96, "end": 51.92, "text": " and then you do arbitrary permutation and you get something like this and you still want to preserve"}, {"start": 51.92, "end": 57.120000000000005, "text": " the the performance that the agent had previously. So obviously if we took something like the DQN"}, {"start": 57.12, "end": 63.28, "text": " agent that's one of the most famous and one of the first successful like RL agents that were"}, {"start": 63.28, "end": 68.32, "text": " able to play these types of games, if you know how it works you know that it's going to fail"}, {"start": 68.32, "end": 74.8, "text": " miserably here because if you learn how to play on this observation it's using CNN in the background"}, {"start": 74.8, "end": 81.6, "text": " remember and so it's going to completely crash here because the like initial structure the original"}, {"start": 81.6, "end": 87.52, "text": " structure the spatial structure that this observation had is not is no longer present here so it's going"}, {"start": 87.52, "end": 91.83999999999999, "text": " to fail. If you haven't watched my DQN video do check it out I'm going to link it somewhere here."}, {"start": 92.72, "end": 98.47999999999999, "text": " So how they managed to achieve this is there are two components to this. So the first component is"}, {"start": 98.47999999999999, "end": 105.36, "text": " this they borrow ideas from this self-organization literature and the second idea is to add the"}, {"start": 105.36, "end": 112.16, "text": " permutation invariance like to bake it in into the model and that way they achieve that this"}, {"start": 112.16, "end": 118.96, "text": " this behavior I just explained. So let's see what I see here. Numerous studies have demonstrated"}, {"start": 118.96, "end": 124.72, "text": " that humans can adapt to changes in sensory inputs even when they are fed into the wrong channels"}, {"start": 124.72, "end": 130.56, "text": " but difficult adaptations such as learning to see by interpreting visual information emitted from a"}, {"start": 130.56, "end": 136.64000000000001, "text": " grid of electrodes placed on one's tongue or learning to ride a backward bicycle require months"}, {"start": 136.64000000000001, "end": 143.6, "text": " of training to achieve mastery. So I mean just the pure fact that humans are able to do this is fairly"}, {"start": 143.6, "end": 149.04, "text": " mind-blowing. So this first thing learning to see through the tongue is you basically have like a"}, {"start": 149.04, "end": 154.64000000000001, "text": " camera I guess and that camera approximately contains the same information as what your eye"}, {"start": 154.64000000000001, "end": 158.64000000000001, "text": " sensor would receive and so you just take the electrode so the information coming from that"}, {"start": 158.64, "end": 165.2, "text": " RGB camera you place it on someone's tongue and the like neural plasticity of the brain does the"}, {"start": 165.2, "end": 171.76, "text": " rest after obviously long like multiple months of training but you can learn how your brain can"}, {"start": 171.76, "end": 177.92, "text": " learn and your neurons can adapt like to form a circuit that can understand the image even though"}, {"start": 177.92, "end": 181.92, "text": " it's coming from a different channel so that's like tongue so that's something called sensory"}, {"start": 181.92, "end": 186.88, "text": " substitution. Like check out this video from getting smarter every day where you have this"}, {"start": 186.88, "end": 191.35999999999999, "text": " backward bike and the goal that the idea is that the input observation is kind of permuted that"}, {"start": 191.35999999999999, "end": 196.48, "text": " means that you try to turn the bar left but the wheel turns right so you have this discrepancy"}, {"start": 196.48, "end": 201.44, "text": " this this shuffling that happened and all of a sudden nobody can actually ride that bike even"}, {"start": 201.44, "end": 205.76, "text": " though you know how to to ride the normal bike and so I guess there is an analogy between that and"}, {"start": 205.76, "end": 211.04, "text": " this thing here it's just that this is a bit more complex and you can imagine we couldn't be able"}, {"start": 211.04, "end": 215.84, "text": " to play this without a severe like a lot of training. So check out this short clip. Where I"}, {"start": 215.84, "end": 220.32, "text": " work the welders are geniuses and they like to play jokes on the engineers. He had a challenge"}, {"start": 220.32, "end": 224.88, "text": " for me. He had built a special bicycle and he wanted me to try to ride it. He had only changed"}, {"start": 224.88, "end": 229.84, "text": " one thing when he turned the handlebar to the left the wheel goes to the right when you turn it to the"}, {"start": 229.84, "end": 234.08, "text": " right the wheel goes to the left. I thought this would be easy so I hopped on the bike ready to"}, {"start": 234.08, "end": 238.56, "text": " demonstrate how quickly I could conquer this. And here he is ladies and gentlemen Mr. Justin Sandlin"}, {"start": 239.2, "end": 241.2, "text": " first attempt riding the bicycle."}, {"start": 241.2, "end": 246.72, "text": " All right so the faster I go the better."}, {"start": 248.72, "end": 254.32, "text": " Yeah yeah I couldn't do it. You can see that I'm laughing but I'm actually really frustrated."}, {"start": 254.32, "end": 260.8, "text": " In this moment I had a really deep revelation. My thinking was in a rut. This bike revealed a"}, {"start": 260.8, "end": 266.32, "text": " very deep truth to me. I had the knowledge of how to operate the bike but I did not have the"}, {"start": 266.32, "end": 272.15999999999997, "text": " understanding. Okay so now the question is can we do better and create AI systems that can rapidly"}, {"start": 272.15999999999997, "end": 278.96, "text": " adapt to sensory substitutions without the need to be retrained. Can we avoid the need to"}, {"start": 278.96, "end": 284.24, "text": " retrain the network in order to complete this task. So even humans need months to adapt to this"}, {"start": 284.24, "end": 290.32, "text": " novel task. So this paper is in a way trying to go even above the human intelligence at least in"}, {"start": 290.32, "end": 296.64, "text": " this very particular domain. One more interesting idea I want to like read out here and then"}, {"start": 296.64, "end": 302.56, "text": " I'm gonna dig into the paper is this. In complex systems we often observe complex global behavior"}, {"start": 302.56, "end": 307.12, "text": " emerge from a collection of agents interacting with each other in their environment with each"}, {"start": 307.12, "end": 312.96, "text": " individual agent acting only on locally available information without knowing the full picture."}, {"start": 312.96, "end": 318.56, "text": " So there are systems such as they mentioned here an implementation of these systems such as cellular"}, {"start": 318.56, "end": 324.16, "text": " automata where basically all of the cells all the components of the system only have access to a"}, {"start": 324.16, "end": 329.76, "text": " local information and then they are able to somehow communicate and form this global representation"}, {"start": 329.76, "end": 349.59999999999997, "text": " this global complex behavior. If you don't know what cellular automata is check out this short video."}, {"start": 359.76, "end": 384.24, "text": " So basically every single cell consults the neighbors and uses their information to update"}, {"start": 384.24, "end": 390.48, "text": " its own information and every single cell does that for itself and then you have this global"}, {"start": 390.48, "end": 396.96000000000004, "text": " like behavior emerging which is pretty amazing. So a similar idea was applied in this paper basically"}, {"start": 396.96000000000004, "end": 404.16, "text": " every single one of these patches will be fed into a unique neural network all of them will be"}, {"start": 404.16, "end": 410.08, "text": " identical the ways will be shared but like still we have like a single neural network will have"}, {"start": 410.08, "end": 415.52, "text": " access to only like a local information so for to a single patch and we somehow need to integrate"}, {"start": 415.52, "end": 420.32, "text": " that information into a global representation which can then be fed into some policy network"}, {"start": 420.32, "end": 425.12, "text": " of an RL agent and then the agent can learn how to play this thing. So the second important component"}, {"start": 425.12, "end": 429.36, "text": " you want to bake in into this model is that no matter the permutation of the patches in the case"}, {"start": 429.36, "end": 434.96, "text": " of these visual games you want to have the same global representation so that means we need to"}, {"start": 434.96, "end": 440.23999999999995, "text": " have some kind of permutation invariance baked in. The way this paper achieves it is using some the"}, {"start": 440.23999999999995, "end": 445.12, "text": " idea from the set transformer paper. Now let me let me break it down and explain all of that."}, {"start": 445.68, "end": 452.32, "text": " Okay so here it is. So the set transformer cleverly replaced Q and again I'm gonna assume you know what"}, {"start": 452.88, "end": 458.32, "text": " QKV is it's a like a terminology from the original transformer paper if you're not familiar with it"}, {"start": 458.32, "end": 463.84, "text": " just check out my transformer video. So they replaced the Q with a set of learnable seed"}, {"start": 463.84, "end": 469.91999999999996, "text": " vectors so it's no longer a function of input X thus enabling the output to become PI or permutation"}, {"start": 469.91999999999996, "end": 477.52, "text": " invariant. So before I dig into this chart here and explain how exactly did they accomplish"}, {"start": 478.08, "end": 481.2, "text": " like the permutation invariance let me quickly explain the difference between"}, {"start": 482.08, "end": 488.23999999999995, "text": " permutation equivariance and permutation invariance. So they mentioned here using these"}, {"start": 488.24, "end": 494.32, "text": " like formulas basically this property needs to be abated if you want to have this is by definition"}, {"start": 494.32, "end": 499.84000000000003, "text": " what permutation invariance is all about. So you have some input X you do some permutation over"}, {"start": 499.84000000000003, "end": 505.28000000000003, "text": " that input X so that's you can treat it as a list of tokens and then once you index it with this S"}, {"start": 505.28000000000003, "end": 511.92, "text": " S is just a permutation of this list here and you basically index X which means which is a maybe a"}, {"start": 511.92, "end": 517.52, "text": " non-Pi notation I don't know if this is like I've never seen this notation in math notebooks but"}, {"start": 517.52, "end": 525.92, "text": " basically what you do here is you're permuting the input X according to this like list S and once you"}, {"start": 525.92, "end": 531.68, "text": " pass that through function F you want to make sure that it's the same as the original like as passing"}, {"start": 531.68, "end": 536.96, "text": " the original input. So no matter the permutation you apply to the input you get the same output"}, {"start": 536.96, "end": 542.48, "text": " that's the permutation invariance and on the other hand we have and as you can see here the interesting"}, {"start": 542.48, "end": 547.6800000000001, "text": " thing here is that the input space dimensionality does not have to be the same as the output space"}, {"start": 547.6800000000001, "end": 553.28, "text": " dimensionality. On the other hand if we have permutation actually variance the those two have"}, {"start": 553.28, "end": 558.5600000000001, "text": " to match so we we gotta have and both in the input space as well as in the output space. So here is"}, {"start": 558.5600000000001, "end": 564.16, "text": " the formula for for permutation actually variance as you can see here so now once we permute the"}, {"start": 564.16, "end": 569.9200000000001, "text": " input what happens is because we have the equivalent part basically the output is going to permute in"}, {"start": 569.92, "end": 575.36, "text": " the same manner. So the original transformer paper had exactly this property and all of the"}, {"start": 575.36, "end": 580.24, "text": " transformers except for the set transformer have this property. So basically let me try and explain"}, {"start": 580.24, "end": 588.56, "text": " it really quickly. So let's say we have three input tokens so a red token we have a green token and"}, {"start": 588.56, "end": 595.12, "text": " we have a blue token. So we pass them into the transformer layer so we have like a black box here"}, {"start": 595.12, "end": 601.84, "text": " this is going to do the transformer magic and out comes some novel representations. So I'll just"}, {"start": 601.84, "end": 608.88, "text": " gonna denote them as this double bar red and then so let me take the green one double bar green so"}, {"start": 608.88, "end": 615.76, "text": " that's some novel representation and finally the blue one. So the the reason why transformers are"}, {"start": 615.76, "end": 620.88, "text": " permutation equivalent is because if we imagine I'm just gonna permute this input now like this."}, {"start": 620.88, "end": 627.6, "text": " So we're gonna have red here we're gonna have green here and we're gonna have blue remain on"}, {"start": 627.6, "end": 633.12, "text": " the same position. So what is now going to happen is that we're just going to permute the output the"}, {"start": 633.12, "end": 638.4, "text": " same way as we permuted the input. So that's what this formula here is telling us. Basically that's"}, {"start": 638.4, "end": 641.76, "text": " what transformers are going to do by default. So we're gonna have the following thing we're gonna"}, {"start": 641.76, "end": 650.64, "text": " have green will be here red will be here and blue will be here. Okay so let me make the permutation"}, {"start": 650.64, "end": 656.0, "text": " explicit. So we have we have this is the original like ordering of the inputs one two three what we"}, {"start": 656.0, "end": 661.92, "text": " did here is we put two here because two corresponds to green one then we had one then we had three. So"}, {"start": 661.92, "end": 668.16, "text": " this is the permutation we applied to the input. So now what this formula tells us is that passing"}, {"start": 668.16, "end": 675.52, "text": " this permuted input through H and this is H in our case so I just you know the T, T's are H. So passing"}, {"start": 675.52, "end": 682.96, "text": " this in the input so this novel permutation so that means this thing here should be the same as"}, {"start": 682.96, "end": 688.8, "text": " passing as passing the original input so that means this one here passing it as the so the output of"}, {"start": 688.8, "end": 694.24, "text": " that thing is going to be this thing and then we apply the the permutation so that means we"}, {"start": 694.24, "end": 698.3199999999999, "text": " permute this thing like this. So we're going to take the the second output which is this and put"}, {"start": 698.3199999999999, "end": 702.88, "text": " it in the first place we're gonna take the first output and put it in the second place"}, {"start": 702.88, "end": 708.72, "text": " we're gonna take the third output and just leave it where it is and as you can see we get exactly"}, {"start": 708.72, "end": 714.72, "text": " this output here. So hopefully this was clear maybe a bit confusing but yeah this is the this is the"}, {"start": 714.72, "end": 720.64, "text": " permutation equivariance. So now what we're trying to accomplish is instead of having this the outputs"}, {"start": 721.36, "end": 726.48, "text": " being permuted the same way as the inputs we want to have them remain fixed. So that means for example"}, {"start": 726.48, "end": 733.28, "text": " we want to have this type of arrangement no matter how we permute the input. In this case we'll have"}, {"start": 733.28, "end": 738.8000000000001, "text": " like three factorial type like possibilities to permute the input and we'll always have this"}, {"start": 738.8000000000001, "end": 743.9200000000001, "text": " output so that will be the permutation invariance. Now one more important detail between permutation"}, {"start": 743.9200000000001, "end": 750.48, "text": " equivariance and invariance is that in the case of equivariance the like the number of input tokens"}, {"start": 750.48, "end": 754.8000000000001, "text": " and I'm going to use the terminology from transformers has to be the same as the number"}, {"start": 754.8, "end": 759.8399999999999, "text": " of output tokens. Whereas once you have the permutation invariance you can have for example"}, {"start": 759.8399999999999, "end": 764.8, "text": " arbitrary number of tokens here like n and you can still keep this maybe I don't know like five"}, {"start": 764.8, "end": 770.24, "text": " whatever. So that's another important detail and we'll soon see why that is important basically"}, {"start": 771.4399999999999, "end": 778.24, "text": " we can some of the patches of the input patches we can kind of omit them and the agent will still"}, {"start": 778.24, "end": 785.04, "text": " be able to play the game. Nice now let me try to explain how the set transformer idea comes along"}, {"start": 785.04, "end": 793.28, "text": " here. So this is the input token these are the input tokens so we have in the case of the image"}, {"start": 793.28, "end": 800.72, "text": " all of these will be just like a single patch so this ot means observation at time t and we are"}, {"start": 800.72, "end": 806.88, "text": " taking patch number one and we are feeding them into these sensory neurons. So as I said these are"}, {"start": 806.88, "end": 811.2, "text": " going to be shared so that's the idea borrowed from the self-organization literature you basically"}, {"start": 811.2, "end": 817.28, "text": " have like this network weights are going to be shared across every single one of sensory neurons"}, {"start": 817.28, "end": 822.72, "text": " and basically as you can see every sensory neuron will have as an input just the local information"}, {"start": 822.72, "end": 828.8, "text": " so just in the case of images just a couple of pixels in that single patch. So what will happen"}, {"start": 828.8, "end": 835.92, "text": " is all of them are going to generate the keys and the value vectors so that's your usual transformer"}, {"start": 835.92, "end": 842.3199999999999, "text": " thing and then instead of generating query vectors as well we decouple that part we decouple this part"}, {"start": 842.3199999999999, "end": 847.52, "text": " as you can see from the input so that means that this Q matrix is just going to be like a fixed"}, {"start": 847.52, "end": 852.56, "text": " embedding like a like an embedding table you can think of it this way like imagine we have just"}, {"start": 852.56, "end": 858.4, "text": " three vectors here okay and that means that if you know how transformers work that means that"}, {"start": 858.4, "end": 864.4, "text": " we're always going to have three vectors at the output here so this message will have constant"}, {"start": 864.4, "end": 870.16, "text": " format of three vectors like this no matter the number of inputs so no matter the number of patches"}, {"start": 870.16, "end": 875.1999999999999, "text": " we input we're always going to have three vectors as the output because that's dictated by the Q"}, {"start": 875.1999999999999, "end": 880.16, "text": " matrix. Let me now try and explain why this thing is permutation invariant so if we take this"}, {"start": 880.16, "end": 885.6, "text": " particular query vector this one here and we do a dot product with all of the keys here so we'll"}, {"start": 885.6, "end": 893.36, "text": " end up with with coefficients alpha 1 alpha 2 all the way through alpha n okay and we're going to"}, {"start": 893.36, "end": 905.36, "text": " have value vectors basically v1 v2 all the way through vn and as you remember as recall we just"}, {"start": 905.36, "end": 911.84, "text": " do multiplication between the attention coefficients and the value vectors we add them up and we form"}, {"start": 911.84, "end": 918.0, "text": " the message here so because of the addition that's that's the part that enables us the permutation"}, {"start": 918.0, "end": 924.08, "text": " invariance so now imagine I just permute I randomly permute the input observation so what will happen"}, {"start": 924.08, "end": 931.12, "text": " is that the key vectors will be randomly permuted as well as the value vector so maybe so then after"}, {"start": 931.12, "end": 937.28, "text": " we apply this query after we do the dot product between query and those new keys and new value"}, {"start": 937.28, "end": 942.16, "text": " vectors I mean the new order of the keys and values we'll just have this this the following"}, {"start": 942.16, "end": 948.64, "text": " thing so basically let's imagine we have something like this alpha 2 alpha 1 all the way through alpha"}, {"start": 948.64, "end": 953.92, "text": " n and because the value vectors are going to be permuted the same in the same manner we'll just"}, {"start": 953.92, "end": 961.1999999999999, "text": " have v2 v1 all the way through vn and now again because we're just doing multiplication between"}, {"start": 961.1999999999999, "end": 966.4, "text": " these and just adding them up we're going to end up with the same result and that's why we have"}, {"start": 966.4, "end": 973.4399999999999, "text": " permutation invariance so basically by doing this we bake in by design we bake in this permutation"}, {"start": 973.4399999999999, "end": 978.56, "text": " invariance property into our neural network so we don't have you do not have to learn how to handle"}, {"start": 978.56, "end": 984.56, "text": " different permutations of the inputs okay and basically on a high level what will happen with"}, {"start": 984.56, "end": 989.68, "text": " all of these agents and they tested these agents on four different games is we're going to input"}, {"start": 989.68, "end": 993.84, "text": " the observation we're going to input the previous action we're going to pass them through this"}, {"start": 993.84, "end": 998.4, "text": " attention neuron so this is this whole pipeline we just saw here we're going to form the message"}, {"start": 998.4, "end": 1003.0400000000001, "text": " we're going to have some policy function here and we're going to output actions the next action"}, {"start": 1003.0400000000001, "end": 1010.0, "text": " that will that we will use to interact with the environment okay having explained this like how"}, {"start": 1010.0, "end": 1015.52, "text": " this thing works I'm just going to make a small remark here so these here are sensory neurons and"}, {"start": 1015.52, "end": 1020.72, "text": " the whole thing is called attention neuron so I'm just going to go back to the title the title name"}, {"start": 1020.72, "end": 1025.84, "text": " was the sensory neuron as a transformer and I think this part should be attention neuron I guess"}, {"start": 1025.84, "end": 1033.6000000000001, "text": " because that's where the transformer like attention happens and not on the sensory neuron level"}, {"start": 1033.6000000000001, "end": 1038.32, "text": " but yeah and anyways a super small net I may be off here let's get back to the paper okay"}, {"start": 1038.88, "end": 1046.24, "text": " so quick notice here so everything I just explained visually is just captured here"}, {"start": 1046.24, "end": 1053.28, "text": " algebraically basically what will happen is every one of these sensory neurons is gonna from the"}, {"start": 1053.28, "end": 1058.72, "text": " observation is going to create its own key is going to create its own value and as you can see"}, {"start": 1058.72, "end": 1065.76, "text": " here the inputs are a bit different the key function will be accepting the observation so this"}, {"start": 1065.76, "end": 1072.32, "text": " like the ith component as well as the previous action whereas the value like functions will only"}, {"start": 1072.32, "end": 1078.8799999999999, "text": " be accepting the observations and now I guess the reason for that is you do not want to bias"}, {"start": 1078.8799999999999, "end": 1085.12, "text": " the like current action with the previous one because remember how this thing works is you're"}, {"start": 1085.12, "end": 1090.6399999999999, "text": " gonna just have in the this message which is just going to contain like a sum a weighted sum of"}, {"start": 1090.6399999999999, "end": 1097.2, "text": " value vectors and so these value vectors will not have like the previous section like incorporated"}, {"start": 1097.2, "end": 1105.04, "text": " into them and so that means we won't won't be biasing the the current action at this time step"}, {"start": 1105.04, "end": 1109.1200000000001, "text": " so I may be completely wrong here but I think that's that's the reason they omitted the"}, {"start": 1109.68, "end": 1115.52, "text": " last action for the value function here finally this is just the transformer formalism the"}, {"start": 1115.52, "end": 1120.96, "text": " important thing to notice here is that the q does not have any input and that's the way we"}, {"start": 1120.96, "end": 1127.04, "text": " achieve the permutation invariance and the second thing is they decoupled like the q matrices with"}, {"start": 1127.04, "end": 1132.24, "text": " this wq's which is not usually done because once you apply the wq's you actually get the queries"}, {"start": 1132.24, "end": 1137.84, "text": " and the keys and the values so here they just decouple this to to make this formalism a bit more"}, {"start": 1137.84, "end": 1142.24, "text": " clear I guess but again if you understood the what I explained here you don't need to worry"}, {"start": 1142.24, "end": 1149.76, "text": " about the formulas okay so a couple more things they have four experiments they have these non"}, {"start": 1149.76, "end": 1156.0, "text": " vision continuous control tasks like the this this like card pull swing up and they also have"}, {"start": 1156.0, "end": 1160.8, "text": " vision based tasks like the pong game and the car race game and I'm gonna switch to blog and show"}, {"start": 1160.8, "end": 1166.16, "text": " you the animations and then I'm gonna return back to the paper and show you the the results they got"}, {"start": 1166.16, "end": 1172.48, "text": " okay here we are basically we have a like this card pull problem it's a very famous toy problem"}, {"start": 1172.48, "end": 1179.36, "text": " in the RL field you're trying to just basically make sure that this card is at the x equals zero"}, {"start": 1179.36, "end": 1184.48, "text": " position along this x-axis and you're trying to make sure that the theta angle equals zero that"}, {"start": 1184.48, "end": 1190.0, "text": " means that this bar the pole is vertical and so what will now happen as you can see here these"}, {"start": 1190.0, "end": 1195.1200000000001, "text": " are the observations so we have five we'll have five sensory neurons well one will be processing"}, {"start": 1195.1200000000001, "end": 1201.52, "text": " at this point like this theta dot which is angular velocity then we have cosine and sine of theta"}, {"start": 1201.52, "end": 1207.52, "text": " we have the x velocity and we have the x position because of the way this agent is implemented and"}, {"start": 1207.52, "end": 1213.28, "text": " that's using LSTM for the sensory neurons that means once I click shuffle observation it will"}, {"start": 1213.28, "end": 1218.8799999999999, "text": " take some time for the hidden state of the LSTM to recover and understand which input is parsing"}, {"start": 1218.8799999999999, "end": 1223.92, "text": " and that's what we'll have a small transition once I hit this and then we'll the agent will"}, {"start": 1223.92, "end": 1228.16, "text": " still manage to to achieve the performance and to balance the pole so let me let me click shuffle"}, {"start": 1228.16, "end": 1234.96, "text": " observations okay so nothing happened there was just some small glitch there let me touch it again"}, {"start": 1234.96, "end": 1241.76, "text": " so this time it failed but it's gonna manage to balance it out again so that's one example"}, {"start": 1241.76, "end": 1246.72, "text": " the non-visual task they were they were using their experiments there is this is the second"}, {"start": 1247.36, "end": 1252.64, "text": " experiment they've done and that's using the pong game and as you can see here aside from shuffling"}, {"start": 1252.64, "end": 1258.48, "text": " they also mask certain patches like certain patches out and you can imagine this would be"}, {"start": 1258.48, "end": 1264.8, "text": " super hard for a human to play and the agent actually manages to to to play it really well"}, {"start": 1264.8, "end": 1270.96, "text": " masking out the the patches is possible because of the m versus n so the input and the output space"}, {"start": 1270.96, "end": 1275.44, "text": " do not have to be the same that means we can change the number of like components of the"}, {"start": 1275.44, "end": 1280.88, "text": " observation at the input and we're still going to have the same message at the output and the policy"}, {"start": 1280.88, "end": 1286.0, "text": " will be able to handle it so that's why we can handle these like patches so yeah aside from that"}, {"start": 1286.0, "end": 1294.08, "text": " we have like the car race game and they showed that they can that their agent can actually handle"}, {"start": 1294.08, "end": 1299.28, "text": " different backgrounds even though the agent never saw these different backgrounds during the training"}, {"start": 1299.28, "end": 1305.44, "text": " and the reason it can achieve that is because the representation that this like model like"}, {"start": 1305.44, "end": 1311.92, "text": " constructs is basically abstracts out the background as a not important information to play this game"}, {"start": 1311.92, "end": 1317.04, "text": " but we'll get back to to those details a bit later this is just for you to understand and have a clear"}, {"start": 1317.04, "end": 1322.0, "text": " picture of the tasks they were using in this in in their experiments now quickly getting back to"}, {"start": 1322.0, "end": 1328.56, "text": " the paper let's digest the results they got so this is the carpool like a static image and again"}, {"start": 1328.56, "end": 1334.08, "text": " kudos to the authors for making this interactive demos in their blog i think that's the the way"}, {"start": 1334.08, "end": 1340.8, "text": " forward because pdfs are such an obsolete format in my honest opinion but yeah um okay so let's see"}, {"start": 1340.8, "end": 1345.76, "text": " the results they achieve they have two baselines first is fully connected neural network with five"}, {"start": 1345.76, "end": 1350.0, "text": " observations the second one is fully connected neural network with 10 observations so these"}, {"start": 1350.0, "end": 1356.24, "text": " fives are recalled that we had like the x the uh the velocity we had the theta the angular velocity"}, {"start": 1356.24, "end": 1361.36, "text": " etc so those are the five observations these baselines will be just you just input the five"}, {"start": 1361.36, "end": 1367.36, "text": " observations into a fully connected layer you get a intermediate hidden representation you you pass"}, {"start": 1367.36, "end": 1373.68, "text": " it to another fully connected layer and out comes the output so these are these baselines and the"}, {"start": 1373.68, "end": 1379.44, "text": " this ours is just instead of using here a feed forward network we're going to use this attention"}, {"start": 1379.44, "end": 1384.48, "text": " neuron we just developed so this is going to be attention neuron component here attention"}, {"start": 1384.48, "end": 1389.6, "text": " neuron okay so these are the models we are comparing with and as you can see here on the"}, {"start": 1389.6, "end": 1395.84, "text": " five observations it's a bit it's lagging a little bit behind the baseline and obviously we have"}, {"start": 1395.84, "end": 1401.84, "text": " uh like no answer here because we cannot plug in five observations into this neural network"}, {"start": 1401.84, "end": 1407.1200000000001, "text": " which is expecting 10 observations so that's the uh the another con of these baselines they are not"}, {"start": 1407.1200000000001, "end": 1412.4, "text": " flexible they are not adaptable to the variation of the number of inputs so the reason there is a"}, {"start": 1412.4, "end": 1417.3600000000001, "text": " small lag here is because they're using lstm so it takes some time to to build up to understand"}, {"start": 1417.3600000000001, "end": 1424.0, "text": " the input that the that particular sensory neuron is is parsing and uh now makes me wonder whether"}, {"start": 1424.0, "end": 1430.0800000000002, "text": " we can just use uh instead of lstm's just use the fully connected network and achieve better results"}, {"start": 1430.0800000000002, "end": 1436.0, "text": " here i'm not sure i saw in the paper like some ablation like that one and i guess it should work"}, {"start": 1436.0, "end": 1442.3200000000002, "text": " since this thing is working already i don't see why this wouldn't work but yeah anyways uh the the"}, {"start": 1442.32, "end": 1446.72, "text": " the point here is that once we shuffle the observations uh this thing completely fails"}, {"start": 1447.28, "end": 1454.3999999999999, "text": " and the attention neuron a model uh doesn't have any problems dealing with shuffled uh like inputs"}, {"start": 1454.3999999999999, "end": 1458.96, "text": " then uh expanding to 10 observations which is proving the point that this model can handle"}, {"start": 1458.96, "end": 1463.76, "text": " varying number of inputs you can see that result is still like holds here here we don't have an"}, {"start": 1463.76, "end": 1468.8, "text": " answer because this is as i said five observations and we we can only test the 10 observation uh"}, {"start": 1468.8, "end": 1473.44, "text": " fully connected neural network and again it's lagging a little bit behind it uh finally the"}, {"start": 1473.44, "end": 1479.76, "text": " interesting column for me is this one because um instead of adding uh and instead of duplicating"}, {"start": 1479.76, "end": 1486.96, "text": " the the five uh input observations this time they actually uh add random gaussian noise and the the"}, {"start": 1486.96, "end": 1493.84, "text": " query vector somehow learned to ignore that information and only focus on the actual uh"}, {"start": 1493.84, "end": 1500.8799999999999, "text": " signals so basically if i go if i scroll back to this chart uh that means that these query vectors"}, {"start": 1500.8799999999999, "end": 1506.08, "text": " are learned such that even though we have even though that like a half of these observations"}, {"start": 1506.08, "end": 1512.9599999999998, "text": " are like pure noise it will learn how to ignore these and so i guess the the the the corresponding"}, {"start": 1512.9599999999998, "end": 1518.08, "text": " attention coefficients will be super small and so the value vectors will be smaller that were"}, {"start": 1518.08, "end": 1524.0, "text": " emitted from these observations and so the message will be constructed only from the mostly from from"}, {"start": 1524.0, "end": 1529.52, "text": " the useful signal uh parts of the observation and not the noise and that's fairly fairly intriguing"}, {"start": 1529.52, "end": 1534.96, "text": " property that that happened there on this diagram they just visually present how the message so the"}, {"start": 1535.76, "end": 1541.6799999999998, "text": " the uh global representation is the same so basically in the case of a card pool"}, {"start": 1541.68, "end": 1548.0800000000002, "text": " uh this empty vector so let me let me scroll back here so this empty is just a 16 dimensional uh like"}, {"start": 1548.0800000000002, "end": 1555.92, "text": " a vector and they just plot it here so this here you pick one time step and this will be the empty"}, {"start": 1555.92, "end": 1561.92, "text": " at that time step and as you can see here these two plots are the same which is nice because uh"}, {"start": 1561.92, "end": 1566.3200000000002, "text": " here we don't have the observation shuffling and here we have the observation shuffling and the"}, {"start": 1566.3200000000002, "end": 1571.04, "text": " message remains the same which means this is just a visual representation of the of the"}, {"start": 1571.04, "end": 1578.48, "text": " of the permutation invariance property uh okay uh so i already showed you uh the demos and"}, {"start": 1579.28, "end": 1584.8, "text": " there is just they just did a bit more um they were very thorough with these experiments so what"}, {"start": 1584.8, "end": 1589.92, "text": " they did is they did this uh training occlusion ratio they they they changed it from zero to"}, {"start": 1589.92, "end": 1595.44, "text": " 0.9 which means zero means you you don't have any occlusion so all of the the patches are actually"}, {"start": 1595.44, "end": 1602.56, "text": " uh contained contain the signal they are not like blacked out and 0.9 means 90% of these patches will"}, {"start": 1602.56, "end": 1609.44, "text": " be blacked out and so training the agent across that range and then testing it again on various"}, {"start": 1610.4, "end": 1615.28, "text": " ratios of occlusion you can see an interesting pattern emerging here so we definitely see the"}, {"start": 1615.28, "end": 1623.2, "text": " best results for 50% and so as a quick reminder for the game of punk uh 21 is the top score you"}, {"start": 1623.2, "end": 1628.32, "text": " can achieve so you're that you have two paddles here you're playing the game and you can uh once"}, {"start": 1628.32, "end": 1634.48, "text": " the the ball gets past your opponent that's plus one for you and you can get until 21 so because"}, {"start": 1634.48, "end": 1640.72, "text": " these are averages um 21 means you're pretty much constantly winning the game and so as you can see"}, {"start": 1640.72, "end": 1647.3600000000001, "text": " here until we get to approximately 50% of occlusion in the test we are performing really well uh and"}, {"start": 1647.3600000000001, "end": 1651.6000000000001, "text": " then it drops it seems that the agents are struggling if we have too much occlusion during"}, {"start": 1651.6, "end": 1656.7199999999998, "text": " training but this this one this one is they mentioned this example here and it's interesting"}, {"start": 1656.7199999999998, "end": 1663.12, "text": " because when you when you basically train with 80% occlusion and then you show all of the patches"}, {"start": 1663.12, "end": 1668.1599999999999, "text": " in the test time you can see that it's still on pair with the opponent meaning that on average"}, {"start": 1668.1599999999999, "end": 1672.8799999999999, "text": " it's even a bit better than the opponent when you have all of the patches which means it can"}, {"start": 1674.3999999999999, "end": 1680.9599999999998, "text": " leverage the additional information and generalize to it even though it did not have it during the"}, {"start": 1680.96, "end": 1686.96, "text": " training time okay they also analyze the representations they get from this attention"}, {"start": 1686.96, "end": 1692.88, "text": " neuron so we just what they do is they take the empty vector they just apply t-sne and they just"}, {"start": 1692.88, "end": 1699.3600000000001, "text": " project the empty to two-dimensional space so if we take uh like outputs which are close to each"}, {"start": 1699.3600000000001, "end": 1705.92, "text": " other so empty is projected nearby in the 2d space we can see that the corresponding inputs that led"}, {"start": 1705.92, "end": 1711.6000000000001, "text": " to that empty being projected here are fairly similar so here you can see that this uh i guess"}, {"start": 1711.6000000000001, "end": 1717.3600000000001, "text": " this is called pad or paddle uh is all the way down and in all of these inputs so basically this"}, {"start": 1717.3600000000001, "end": 1722.88, "text": " is a single input this whole row is a single input because again it would be helpful if you knew how"}, {"start": 1722.88, "end": 1727.6000000000001, "text": " dqm works especially when you're dealing with these Atari games what you do you stack four"}, {"start": 1727.6000000000001, "end": 1735.04, "text": " consecutive frames to avoid having this uh perceptual aliasing whereby you do not know if you see a"}, {"start": 1735.04, "end": 1739.36, "text": " single image whether the ball is going in this direction or that direction or that direction so"}, {"start": 1739.36, "end": 1745.76, "text": " you do not know the state once you concatenate couple frames you have uh like a less uh like a"}, {"start": 1745.76, "end": 1751.92, "text": " vagueness let's go that way in your input so you can see that the inputs are are visually and"}, {"start": 1751.92, "end": 1757.28, "text": " semantically similar and they are grouped here into the same uh like part of this 2d space here"}, {"start": 1757.28, "end": 1762.08, "text": " you can imagine once you have this type of representation the agent can learn how to to"}, {"start": 1762.08, "end": 1767.4399999999998, "text": " make good decisions and play play play the game of punk okay um i mentioned this showed you i showed"}, {"start": 1767.4399999999998, "end": 1774.24, "text": " you this in the blog uh basically uh the cool thing is that the uh like our this model the"}, {"start": 1774.24, "end": 1779.1999999999998, "text": " attention neuron manages to to cope with different types of backgrounds so they they tried four"}, {"start": 1779.1999999999998, "end": 1785.28, "text": " different backgrounds here uh plus the shuffling and they showed that the like agents still can't"}, {"start": 1785.28, "end": 1790.0, "text": " play the game because it's ignoring uh that irrelevant information which is the background"}, {"start": 1790.0, "end": 1796.16, "text": " in this case and they plotted on this diagram here the attention so how they construct these"}, {"start": 1796.16, "end": 1800.88, "text": " representations is remember these are patches this is going to be your input it you're going"}, {"start": 1800.88, "end": 1807.2, "text": " to pass it to the sensory neuron and it's going to emit the key and so once you you take your query"}, {"start": 1807.2, "end": 1813.36, "text": " you just do dot product with all the keys and you can see that these here uh get the most attention"}, {"start": 1813.36, "end": 1818.56, "text": " which is basically the like the the curve the turning so i mean you as humans we can we can"}, {"start": 1818.56, "end": 1823.04, "text": " notice that these are very important because once you take the features that correspond to these"}, {"start": 1823.04, "end": 1829.9199999999998, "text": " patches and you combine them into the message and then you can basically use that feature that novel"}, {"start": 1829.9199999999998, "end": 1835.28, "text": " global representation to make a good decision and that's to turn left in this case similarly here"}, {"start": 1835.28, "end": 1840.56, "text": " even though they swapped the background so the original background is this screen like a grass"}, {"start": 1840.56, "end": 1846.56, "text": " type of thing once you swap it with this random background the the model still learns how to"}, {"start": 1846.56, "end": 1851.2, "text": " uh pay attention to the relevant parts of the image and so it knows when to turn i think that's"}, {"start": 1851.2, "end": 1858.72, "text": " pretty much it uh there is one more interesting thing here and that's uh they not only shuffle the"}, {"start": 1858.72, "end": 1864.56, "text": " input observation they do that dynamically during the the game so every for example t equals 25"}, {"start": 1864.56, "end": 1870.96, "text": " steps they shuffle the input observation and they do that from 25 50 all the way to 500 and no"}, {"start": 1870.96, "end": 1875.44, "text": " reshuffle and we can see the results here are pretty pretty nice so looking at the car racing"}, {"start": 1875.44, "end": 1881.44, "text": " uh the the performance drops somewhat same as for a carpool but for different reasons so here we have"}, {"start": 1881.44, "end": 1886.0800000000002, "text": " the lstm so it takes some time to update the hidden state here they have some different strategy"}, {"start": 1886.0800000000002, "end": 1892.4, "text": " they're using some difference between the the the neighboring frames and so once you do the uh like"}, {"start": 1892.4, "end": 1897.2, "text": " a shuffle that means that that difference is going to have a it's going to introduce an error into"}, {"start": 1897.2, "end": 1902.64, "text": " the model and that's it's going to struggle uh like uh until it stabilizes again so there are a"}, {"start": 1902.64, "end": 1908.3200000000002, "text": " bunch of details they did a very nice and thorough experimentations across different games you can"}, {"start": 1908.3200000000002, "end": 1913.8400000000001, "text": " check out the blog yourself and the interactive demos but basically the main ideas are uh to have"}, {"start": 1913.8400000000001, "end": 1920.64, "text": " this type of sharing uh between sensory neurons and to add the the set transformer idea so that"}, {"start": 1920.64, "end": 1925.2800000000002, "text": " we have the permutation invariance and so that we can shuffle the observation and still preserve"}, {"start": 1925.2800000000002, "end": 1930.48, "text": " the same empty upon which we are building the policy network so i haven't been focusing obviously"}, {"start": 1930.48, "end": 1936.72, "text": " on the ariel component because that's just um not the the core inside behind this paper they were"}, {"start": 1936.72, "end": 1942.4, "text": " also using evolutionary evolutionary strategy algorithms to basically train the agents but the"}, {"start": 1942.4, "end": 1949.2, "text": " main idea was to test whether the agents can be more robust to permutation to random shuffling of"}, {"start": 1949.2, "end": 1953.52, "text": " the input uh observations so hopefully you found this video useful if you did share it out with a"}, {"start": 1953.52, "end": 1958.08, "text": " friend subscribe this channel join the discord community there is a lot of cool conversations"}, {"start": 1958.08, "end": 1974.0, "text": " happening there people are helping each other out so do join it and until next time bye bye"}]