RMA: Rapid Motor Adaptation for Legged Robots | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover the "Rapid Motor Adaptation for Legged Robots" paper where they introduce this novel technique which allows legged robots to adapt, in real-time, to novel never before seen environments!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://ashish-kmr.github.io/rma-legged-robots/rma-locomotion-final.pdf
✅ Blog: https://ai.facebook.com/blog/ai-now-enables-robots-to-adapt-rapidly-to-changing-real-world-conditions
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Introducing RMA
02:15 Paper overview
04:15 Architecture explained
12:00 Reward shaping
15:30 Curriculum training
17:40 No need to predict env vector
18:50 Results and experiments

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#reinforcementlearning #robots #adaptation
 AI now enables robots to adapt rapidly to changing real-world conditions. So this research just came out yesterday. It's from Facebook, CMU, and Berkeley AI research, and introduced this thing called rapid motor adaptation, which is an awesome new technique. Let's call it that way. It leverages reinforcement learning plus supervised learning in order to do these rapid adaptations to various novel and unknown environments. As you can see here, so let me just zoom in. This robot is basically trained so the reward function is just move straight ahead and there's like, it's like heavily shaped. We'll see that in a moment. But the whole point I want you to notice here is that this matrix is oiled up, so, so, and the robot never has never seen something like a similar environment like and it hasn't been fine-tuned. Neither has been the simulation calibrated to kind of depict the state of this particular environment. So again, it's trained completely in the simulation. It's using RL, it's using supervised learning, and we'll see the details of the paper in a second. But let me just show you a couple more cool examples. So here again, you can see it's just moving straight ahead, so it's kind of going towards the rock. But that's not the point. You can see it's adaptively, it can kind of go down these rocks and it doesn't kind of fall over. So it's pretty, it looks pretty, much more natural compared to some of the like previous methods we've been seeing. And finally, this one is really cool. So the robot goes over this super hard environment and the leg kind of stucks here. And then after a couple of moments, instead of falling over, it unstucks and it continues walking. So I find this super exciting. Like I can imagine like a huge number of cool applications where these robots could be doing the jobs that humans can't or should not, like some radiation, high radiation areas or whatnot. Again, this can and will probably be used in military purposes, but that's, I guess, a topic for another video. So let me now, having said that, let's jump to the paper and let's see how this thing works. Okay. So again, this is a work by Ashish Kumar, Zipeng Fu, Deepak Pathak and Jitendra Malik. So that's Berkeley folks, CMU and Facebook. And what I want you to notice here is they've been testing this on a wide variety of environments. And interesting thing is written here. So the robot achieved this high success rate despite never having seen unstable or sinking ground, obstructive vegetation or stairs during training. All deployment results are with the same policy. So the same policy without any simulation calibration or real world fine tuning. So usually if you watch my video on OpenAI's robotic hand solving Robux cube, which I super recommend you go ahead and watch. So basically they use this thing called randomize domain adaptation, where the whole point is to, so you have a special environment in which you want to deploy a robot. And in their case, they had a cage, they had this robotic hand. And so there was a really constrained setup. And what they did is they kind of measured all of the relevant information, like where the cameras are positioned, and they constructed the simulation in such a way so that it's calibrated. So the main parameters such as camera position, et cetera, are kind of like baked in into the simulation. Here on the other hand, they don't do that. They have, we'll see like the simulation is just a sampling bunch of randomized terrains, which are hard. And they do some, again, they do some curriculum learning. But like the important thing is they don't do the simulation calibration and they don't have this real world fine tuning. I think that even that OpenAI's paper did this fine tuning whereby you basically capture some real data and then you further fine tune your policy on that real data so that you can better bridge the gap between the simulation and the real world. But here they don't do that and that's super exciting. Okay, let's see how this model works on the high level. So here is the whole pipeline. Basically we have, so we have first, the first part is training, obviously, the second part is deployment, where we are using the policy trained in A in the real world. So the training part itself contains out of, consists out of two parts. So the first part is this policy training phase, whereby we are taking these parameters of the environment such as like terrain height. So for example, if you focus on this robot here, it will basically have, assuming all the four legs are in contact with the ground, you can figure out, you can calculate because you're in a simulation, you have all of the ground truth data, they just calculate the highest peak of those four points and that's one of the like information bits they are using in this ET vector. They also have this motor strength, they have this mass and center of mass information. So that's basically, they're putting certain mass objects on top of this robot's back because they'll be testing it in the same setup later on. And so obviously this thing won't be available during the deployment phase because that's just something that you have in a simulator. And so again, how the whole thing's work is the following. So they have this, this first stage, they train this RL policy based policy PI, and they just use like proximal policy optimization algorithm from OpenAI. So that's PPO. And again, the goal is just to maximize the reward where the reward is super shaped. We'll see that in a couple of minutes, but like for now, let's just focus and see how this works. So they encode this information. So this environmental factor vector using this like n factor encoder, which is a simple MLP. I think they have like three layer MLP and they encode it into this latent extrinsic, like how they call it, extrinsic latent vector. And it kind of contains the compressed information about the environment, about all of this information here. And aside from that, obviously we are like, we are fitting in the current state, which consists out of the joint position, joint velocities, et cetera. They also have this action. So what the actions are is basically you just set the joint position. These are joint position of all of the 12 actuated joints on this robot. And that will be the action. And finally, those actions will be converted to Torx using something called PD controllers. And then you have your usual like RL setup. Basically you have your environment here. So let me kind of abstract this. You have environment. Let's call it E. You have an agent. And what happens is agent sends the action to the environment. And we can see that here. So that's the action here at point at timestamp T. And what comes back is the pretty much the next state. So that's, they depicted that they denoted that as XT as well as reward. So you'll have some intermediate intermediate rewards. So that's your basic generic RL setup. And again, they just use PPO. They train this base policy. They train this encoder in the first phase. And then the important part is this adaptation module. So that's the true like innovation of this paper. They train this thing to be able to infer this latent information. Just looking at the past observations. That's the whole idea. So this time, instead of having a privilege of accessing this information from the simulator, they just pack the last 50 states and actions. They feed that into this adaptation module, which is again, simple MLP. And then they have some one D convolutions. And basically they output this approximation. So ZT hat, which they then minimize using simple MSC loss. So that's your like L2 square loss. And everything else remains the same. We're still feeding in the state. We're still feeding in the last action. And that's it. So one important thing to notice here is that during this phase two, when they're training the only the adaptation module, so obviously the base policies is frozen here. And second important thing is they are using on policy algorithm here. So they're basically feeding in this ZT hat. So the inferred one, even though initially, obviously, because this is randomly initialized, these will be kind of rubbish in the beginning. So what you'll basically be ending up in like bad states. And that will help this network become robust enough so that you can couple like cope with different deployment scenarios. Because if we were feeding the perfect information coming from this, from these environment variables, then it will the basically the adaptation module would only be exposed to good trajectories. And that's not what we want, because we want it to be able to cope even with these bad trajectories when when something goes wrong. So that's the that's that's it in a nutshell. So in the first again, in the first part, they train the base policy. So that's your RL agent, they train this encoder of the environment factors. And they train this in the second stage, they train this adaptation module, which is really important, which learns how to predict the latent, so condensed information about the environment from the past observations only. And interesting thing to notice here is that so they have no vision component in the system. So all of the videos you clips, the clips you saw earlier were based solely on this RL and supervised learning policies. Okay. So deployment, what I do is they obviously feed the the past observations into the adaptation module, it runs on 10 10 hertz, because you have like 5050 past observations here. So it's kind of needs to run a bit slower. And it infers this this ZT hat, which is information about the environment, every now and then. So these two components are pretty much asynchronous. So that means that the base policy will be running on 100 hertz. So so it would be feeding the state and the last action, and it will just be fetching whatever the last update to this, to this vector of this latent vector is. So that assumption kind of is valid because environment won't be changing. Like empirically, they kind of figure out that it won't be changing, they don't need to improve the the frequency of this, and it just works. Okay, having explained that, let's now jump into some details. First thing, so this is, they notice here that the key, the key contribution of the paper is basically, so our novel aspects are the use of a varied terrain generator, and natural reward functions. So by the way, let me just digress here. So as you can see here, this is the terrain, it's totally random, it looks really hard, you can, the legs can get jammed, etc. So the whole point of this is, similarly to the OpenAI's robotic hand, is to train this thing in a bunch of different hard environments, I forgot to mention that, where they sample different parameters such as mass, and like in the OpenAI paper, they sample gravity even. And so the idea is to expose the policy to such a wide range of randomized worlds, so that when you deploy it in a real world, it just looks like another simulation, and that's the whole point. Simulation hypothesis confirmed. So let's continue here. So the aspects are, as I said, varied terrain generator, and natural reward functions, motivated by bioenergetics, which allows us to learn walking policies without using any reference demonstrations. But the truly novel contribution of this paper is the adaptation module, trained in simulation, which makes the rapid motor adaptation possible. Nice. Um, what next? So here is the reward function they're using. I mentioned it's heavily shaped, as you can see here. So the goal is the following. The goal is to, they say it's somewhere here, so the reward function encourages the agent to move forward with a maximum speed, blah blah blah, and penalizes it for jerky and inefficient motions. So basically you can see here, the goal is to maximize the reward, and if we plot this first component, so the whole reward will just be a sum of these 10 components, and if we just plot this thing here, so basically on the x-axis you have vx, and let me just kind of briefly digress here. So this is a robot, just to explain the coordinate system. This is a robot, pretty retarded, okay, but yeah, it works. And this would be the x-axis, y-axis would be, for example, pointing towards us, and these z-axis points upwards. And having said that, so this velocity x is just a component along this x-axis here, and so it basically increases until 0.35, so this value here, and then it just starts getting into the saturation, which means the robot is encouraged only to get to 0.35 and not above. So they, again, penalize, because this is minus, we want to minimize this part, so we want to minimize the sideways movement, so this is, again, y-axis coming out of the screen towards you, so we want to minimize that velocity component there, as well as this angular velocity, and like a bunch of similar stuff, like torque, we want to minimize torque, we want to minimize the joint motion, we want to minimize the slippage of the foot. So basically what this g of t is, they say it here, so blah blah blah, the binary foot contact indicator vector as g, so that's g. So that means you have four legs, and because you do this dyac operation, what you do is you basically get a matrix where you have a binary matrix, so you have maybe zero, one, zero, zero, one. That means two of the legs are currently in the contact with the floor, and only those, we'll only care about those velocities of the foot. So I think, so they say somewhere here, so vf is the velocity of the feet, okay? So that means we want to minimize that velocity, which makes sense, because that means if the velocity is hard, that means that the legs are kind of slipping around, and that's not something we want. We want a stable robot without jerking, without like these movements, like either sideways movements, or here you can see it's also penalized not to go up and down too much. So anyways, I can just imagine how it looked like to shape this thing. Not the best design, I guess, but like I love the results they got, and yeah. In general, it's not recommended to shape the reward like this, but I guess their contribution lied somewhere else, and they, basically in the adaptation procedure, I guess this part is not as important as the fact that they show that their robots can adapt to unknown environments, even being trained in simulation only, that's a huge, huge thing. And they say it here again, this tells us about the fragility of this shaping. We found these reward functions to be critical for learning realistic gates in simulation, so I guess they had to iterate a lot to get this to work. And again, problems in the paradise, they say here, if we naively train our agent with the above reward function, it learns to stay in a place because of the penalty terms on the movement of the joints. So that translates into this. So this Q with the small dot here, if you can see, just denotes the velocities of the joints. So this term of the reward function will encourage the robot not to move at all, and yeah, as you can see here, if we just start with these, because this will be just a weighted sum of these rewards, right? And you'll have some parameters, as you can see here, they just figured out these parameters empirically. So if you just start with these parameters, the thing won't work, the robot won't move. So they do some sort of annealing, they start with super small coefficients, and then during the training, they slowly raise this up. So it's a training curriculum, obviously. And yeah, and they say to prevent the collapse, we start the training with very small penalty coefficients, so very small penalty coefficients, and then gradually increase the strength of these coefficients using a fixed curriculum. We also linearly increase the difficulty of other perturbations such as mass, friction, and motor strength as the training progresses. We don't have any curriculum on the trains and start the training with randomly sampling the train profiles from the same fixed difficulty. Bottom line, certain parts of the environment are under curriculum, certain parts like the terrain itself is always sampled from the same very same difficulty. That was the whole point. Here in this table, they just show the range they've used for certain parameters in the environment like friction, center of mass, etc. So yeah, so again, center of mass is this thing here, they have this robot, they put like an object with mass on top of its back, and they just test how it behaves under these external forces. That's the whole point. And as I mentioned, this is the same thing as with OpenAI's robotic hand. They sample an environment where they'll be sampling uniformly from these ranges, and these ranges as well will be progressively becoming harder and harder as the training progresses, so that's because they are included in the training curriculum. One thing worth mentioning is that they mention it here. Note that instead of predicting ET, so that's the environmental vector, which is the case in typical system identification, we directly estimate extrinsics ZT that only encodes how the behavior should change to correct for the given environment vector ET. So the whole point here is something similar to Mu0. If you watched that video, if you want to check it out, I'll link Mu0 video somewhere here, I made it a while ago, and the whole idea is the following. You only need to encode that information which makes you behave, have a better policy. You don't need to be able to reconstruct from that latent, so in this case, this environment vector, or in the case of Mu0, the observation space, you just need to have such information upon which you can act and have an optimal policy, and that's it. So you just save some bandwidth on modeling only the information you actually need to behave properly in the environment, and you just discard everything else. Okay, here we have some experiments they've done with certain baselines. So again, they have a bunch of different tasks. They have this uneven foam, they have this upward incline, so they're testing it in a bunch of different mattresses. They're like step up eight centimeters, step up six centimeters, step down 15 centimeters, et cetera. And they just compare it with the other baselines such as the this very same model, just without that adaptation block, as well as this A1. So A1 is just the name of the robot, so A1 is some default controller software that comes as a firmware of this robot when you buy it. And they showed that on all of these environments in these tables, just looking at the success metric, TTF, so it's time to fall and distance, RMA pretty much outperforms the other baselines in all of these tasks, as well as in this payload analysis, we have these diagrams where we are on the x-axis, we are increasing the weight of this thing of the payload on the robot's back, and we can see how different baselines behave, and we can see that the success drops really slowly with this RMA approach, whereas with the baselines, it drops pretty significantly. And worth mentioning is that the robot itself has 12 kilos, which means that at this point on x-axis, where basically the robot is carrying the same amount of weight as its own body weight on its own back. So that's pretty cool. So here they have another analysis on this particular environment with the mattress covered with oil, and what you can notice here is that from this point on, so when it starts stepping on this slippery floor, the environment parameters obviously change, like the friction parameter will change, and we can see that being inferred by the adaptation module exactly here. So we just log, so this latent vector, I think it has like eight dimensions, so they just took Z1 and Z5, so those are two particular features of that latent vector, and they just do, they plot it, and we can see that it basically reacts really quickly to the change of the environment, and that in turns helps the robot. So you can see here, this is the gate patterns. Basically RL means rear left leg, rear right leg, like forward left leg, and forward right leg, and so you can just see these means that so when you have a bar here, that means that the contact is made between this particular leg and the floor, and you can see that things start getting messy here at one point in time, and then the pattern returns to normal even though the robot is still on the environment because of the adaptation module, and also here what I mentioned is that the peaks here, and this is the torque plot as you can see here, the knee torque gets higher after this part here, and that's just a way to visualize what happens internally in that RL agent and adaptation modules, and yeah, pretty awesome behavior. Some final results here are in simulation. They just show a couple more baselines. We have RMA, we have RMA without the adaptation module, we have expert which is using, instead of using the previous observations to figure out the latent environment vector, it just uses the ground truth vectors, and we can see looking at all these metrics like reward, TTF, so time to fall, distance, samples, torque, smoothness, if you remember stuff like torque and stuff like smoothness, and ground impact were like enforced to be minimized by the reward components up there, so let me just, for example, let's take a ground impact or smoothness. If I go there to the reward function, so here we have, so smoothness as you can see here, or ground impact, that means that the force, so F is the, so ground reaction force is at the feet as F, so this part just tells you how elegantly, let's call it that way, the robot is kind of touching the floor, and it should be minimized looking at this because, yeah, because it's a part of the component with a minus prefix here, and just again returning to the table, we can see that the ground impact gets smaller and smaller here, and we can see that the difference between RMA and expert is really small at all of these metrics, so we can see that the ground force is at the feet, so just comparing the success rate, comparing the TTF, comparing reward, comparing distance, and everything else, it seems that RMA is able to infer the latent environment vector almost as well as the ground truth vector, so that's awesome, that means that the supervised learning like knowledge distillation worked, so that's pretty much it, that was a pretty neat idea, awesome results, this has, this can have amazing implications on RL being used in real world environments combined with some other types of learning, paradigms such as supervised learning, and like I guess the like follow-up work to this thing will be integrating vision component and then just deploying it, and next thing you know, Boston Dynamics robots will be running around the city, that's pretty much it, hopefully you like this video, if you did, share it out, comment down below if you have any suggestions for the next papers or whatever you have to say, and until next time, bye bye!
[{"start": 0.0, "end": 5.12, "text": " AI now enables robots to adapt rapidly to changing real-world conditions."}, {"start": 5.12, "end": 12.88, "text": " So this research just came out yesterday. It's from Facebook, CMU, and Berkeley AI research,"}, {"start": 12.88, "end": 19.12, "text": " and introduced this thing called rapid motor adaptation, which is an awesome new technique."}, {"start": 19.12, "end": 24.8, "text": " Let's call it that way. It leverages reinforcement learning plus supervised learning in order to"}, {"start": 24.8, "end": 32.4, "text": " do these rapid adaptations to various novel and unknown environments. As you can see here,"}, {"start": 32.4, "end": 37.68, "text": " so let me just zoom in. This robot is basically trained so the reward function is just move"}, {"start": 37.68, "end": 43.120000000000005, "text": " straight ahead and there's like, it's like heavily shaped. We'll see that in a moment."}, {"start": 43.120000000000005, "end": 48.64, "text": " But the whole point I want you to notice here is that this matrix is oiled up, so,"}, {"start": 48.64, "end": 55.92, "text": " so, and the robot never has never seen something like a similar environment like and it hasn't"}, {"start": 55.92, "end": 64.4, "text": " been fine-tuned. Neither has been the simulation calibrated to kind of depict the state of this"}, {"start": 64.4, "end": 71.2, "text": " particular environment. So again, it's trained completely in the simulation. It's using RL,"}, {"start": 71.2, "end": 75.12, "text": " it's using supervised learning, and we'll see the details of the paper in a second. But let me just"}, {"start": 75.12, "end": 80.32000000000001, "text": " show you a couple more cool examples. So here again, you can see it's just moving straight ahead,"}, {"start": 80.32000000000001, "end": 85.12, "text": " so it's kind of going towards the rock. But that's not the point. You can see it's adaptively,"}, {"start": 85.12, "end": 91.76, "text": " it can kind of go down these rocks and it doesn't kind of fall over. So it's pretty, it looks pretty,"}, {"start": 91.76, "end": 96.56, "text": " much more natural compared to some of the like previous methods we've been seeing."}, {"start": 97.92, "end": 103.84, "text": " And finally, this one is really cool. So the robot goes over this super hard environment and"}, {"start": 103.84, "end": 108.32000000000001, "text": " the leg kind of stucks here. And then after a couple of moments, instead of falling over,"}, {"start": 108.32000000000001, "end": 115.76, "text": " it unstucks and it continues walking. So I find this super exciting. Like I can imagine like a"}, {"start": 115.76, "end": 120.24000000000001, "text": " huge number of cool applications where these robots could be doing the jobs that humans"}, {"start": 121.04, "end": 128.0, "text": " can't or should not, like some radiation, high radiation areas or whatnot. Again, this can and"}, {"start": 128.0, "end": 132.64000000000001, "text": " will probably be used in military purposes, but that's, I guess, a topic for another video."}, {"start": 132.64, "end": 137.44, "text": " So let me now, having said that, let's jump to the paper and let's see how this thing works."}, {"start": 137.44, "end": 143.67999999999998, "text": " Okay. So again, this is a work by Ashish Kumar, Zipeng Fu, Deepak Pathak and Jitendra Malik."}, {"start": 143.67999999999998, "end": 149.92, "text": " So that's Berkeley folks, CMU and Facebook. And what I want you to notice here is they've"}, {"start": 149.92, "end": 156.16, "text": " been testing this on a wide variety of environments. And interesting thing is written here. So the robot"}, {"start": 156.16, "end": 163.35999999999999, "text": " achieved this high success rate despite never having seen unstable or sinking ground, obstructive"}, {"start": 163.35999999999999, "end": 168.0, "text": " vegetation or stairs during training. All deployment results are with the same policy."}, {"start": 168.56, "end": 175.84, "text": " So the same policy without any simulation calibration or real world fine tuning. So usually"}, {"start": 175.84, "end": 182.07999999999998, "text": " if you watch my video on OpenAI's robotic hand solving Robux cube, which I super recommend you"}, {"start": 182.08, "end": 188.64000000000001, "text": " go ahead and watch. So basically they use this thing called randomize domain adaptation,"}, {"start": 188.64000000000001, "end": 194.88000000000002, "text": " where the whole point is to, so you have a special environment in which you want to deploy a robot."}, {"start": 194.88000000000002, "end": 200.0, "text": " And in their case, they had a cage, they had this robotic hand. And so there was a really constrained"}, {"start": 200.0, "end": 204.4, "text": " setup. And what they did is they kind of measured all of the relevant information, like where the"}, {"start": 204.4, "end": 209.60000000000002, "text": " cameras are positioned, and they constructed the simulation in such a way so that it's calibrated."}, {"start": 209.6, "end": 215.44, "text": " So the main parameters such as camera position, et cetera, are kind of like baked in into the"}, {"start": 215.44, "end": 220.07999999999998, "text": " simulation. Here on the other hand, they don't do that. They have, we'll see like the simulation is"}, {"start": 220.07999999999998, "end": 225.6, "text": " just a sampling bunch of randomized terrains, which are hard. And they do some, again, they"}, {"start": 225.6, "end": 231.2, "text": " do some curriculum learning. But like the important thing is they don't do the simulation calibration"}, {"start": 231.2, "end": 236.24, "text": " and they don't have this real world fine tuning. I think that even that OpenAI's paper did this"}, {"start": 236.24, "end": 241.76000000000002, "text": " fine tuning whereby you basically capture some real data and then you further fine tune your"}, {"start": 241.76000000000002, "end": 250.16, "text": " policy on that real data so that you can better bridge the gap between the simulation and the"}, {"start": 250.16, "end": 255.20000000000002, "text": " real world. But here they don't do that and that's super exciting. Okay, let's see how this model"}, {"start": 255.20000000000002, "end": 260.96000000000004, "text": " works on the high level. So here is the whole pipeline. Basically we have, so we have first,"}, {"start": 260.96000000000004, "end": 265.68, "text": " the first part is training, obviously, the second part is deployment, where we are using the policy"}, {"start": 265.68, "end": 273.84000000000003, "text": " trained in A in the real world. So the training part itself contains out of, consists out of two"}, {"start": 273.84000000000003, "end": 280.32, "text": " parts. So the first part is this policy training phase, whereby we are taking these parameters of"}, {"start": 280.32, "end": 286.56, "text": " the environment such as like terrain height. So for example, if you focus on this robot here,"}, {"start": 286.56, "end": 292.96000000000004, "text": " it will basically have, assuming all the four legs are in contact with the ground, you can figure out,"}, {"start": 292.96, "end": 297.91999999999996, "text": " you can calculate because you're in a simulation, you have all of the ground truth data, they just"}, {"start": 297.91999999999996, "end": 304.32, "text": " calculate the highest peak of those four points and that's one of the like information bits they"}, {"start": 304.32, "end": 310.32, "text": " are using in this ET vector. They also have this motor strength, they have this mass and center of"}, {"start": 310.32, "end": 315.35999999999996, "text": " mass information. So that's basically, they're putting certain mass objects on top of this"}, {"start": 315.35999999999996, "end": 322.71999999999997, "text": " robot's back because they'll be testing it in the same setup later on. And so obviously this"}, {"start": 322.72, "end": 327.36, "text": " thing won't be available during the deployment phase because that's just something that you have"}, {"start": 327.36, "end": 332.40000000000003, "text": " in a simulator. And so again, how the whole thing's work is the following. So they have this,"}, {"start": 334.0, "end": 340.88000000000005, "text": " this first stage, they train this RL policy based policy PI, and they just use like proximal policy"}, {"start": 340.88000000000005, "end": 347.44000000000005, "text": " optimization algorithm from OpenAI. So that's PPO. And again, the goal is just to maximize the"}, {"start": 347.44, "end": 353.44, "text": " reward where the reward is super shaped. We'll see that in a couple of minutes, but like for now,"}, {"start": 353.44, "end": 359.52, "text": " let's just focus and see how this works. So they encode this information. So this environmental"}, {"start": 359.52, "end": 365.92, "text": " factor vector using this like n factor encoder, which is a simple MLP. I think they have like"}, {"start": 365.92, "end": 372.08, "text": " three layer MLP and they encode it into this latent extrinsic, like how they call it,"}, {"start": 372.08, "end": 378.0, "text": " extrinsic latent vector. And it kind of contains the compressed information about the environment,"}, {"start": 378.0, "end": 384.24, "text": " about all of this information here. And aside from that, obviously we are like, we are fitting in"}, {"start": 384.24, "end": 389.59999999999997, "text": " the current state, which consists out of the joint position, joint velocities, et cetera."}, {"start": 389.59999999999997, "end": 395.12, "text": " They also have this action. So what the actions are is basically you just set the joint position."}, {"start": 395.12, "end": 400.88, "text": " These are joint position of all of the 12 actuated joints on this robot. And that will be the action."}, {"start": 400.88, "end": 405.92, "text": " And finally, those actions will be converted to Torx using something called PD controllers."}, {"start": 405.92, "end": 409.92, "text": " And then you have your usual like RL setup. Basically you have your environment here. So"}, {"start": 409.92, "end": 417.68, "text": " let me kind of abstract this. You have environment. Let's call it E. You have an agent. And what"}, {"start": 417.68, "end": 423.36, "text": " happens is agent sends the action to the environment. And we can see that here. So that's the action here"}, {"start": 423.36, "end": 430.08, "text": " at point at timestamp T. And what comes back is the pretty much the next state. So that's,"}, {"start": 430.08, "end": 436.32, "text": " they depicted that they denoted that as XT as well as reward. So you'll have some intermediate"}, {"start": 436.32, "end": 442.56, "text": " intermediate rewards. So that's your basic generic RL setup. And again, they just use PPO."}, {"start": 442.56, "end": 447.76, "text": " They train this base policy. They train this encoder in the first phase. And then the important"}, {"start": 447.76, "end": 453.03999999999996, "text": " part is this adaptation module. So that's the true like innovation of this paper. They train this"}, {"start": 453.04, "end": 461.6, "text": " thing to be able to infer this latent information. Just looking at the past observations. That's the"}, {"start": 461.6, "end": 466.96000000000004, "text": " whole idea. So this time, instead of having a privilege of accessing this information from the"}, {"start": 466.96000000000004, "end": 473.12, "text": " simulator, they just pack the last 50 states and actions. They feed that into this adaptation"}, {"start": 473.12, "end": 478.72, "text": " module, which is again, simple MLP. And then they have some one D convolutions. And basically they"}, {"start": 478.72, "end": 486.40000000000003, "text": " output this approximation. So ZT hat, which they then minimize using simple MSC loss. So that's"}, {"start": 486.40000000000003, "end": 492.8, "text": " your like L2 square loss. And everything else remains the same. We're still feeding in the"}, {"start": 492.8, "end": 498.88000000000005, "text": " state. We're still feeding in the last action. And that's it. So one important thing to notice"}, {"start": 498.88000000000005, "end": 504.8, "text": " here is that during this phase two, when they're training the only the adaptation module, so"}, {"start": 504.8, "end": 510.16, "text": " obviously the base policies is frozen here. And second important thing is they are using on policy"}, {"start": 510.16, "end": 516.8, "text": " algorithm here. So they're basically feeding in this ZT hat. So the inferred one, even though"}, {"start": 516.8, "end": 521.2, "text": " initially, obviously, because this is randomly initialized, these will be kind of rubbish in"}, {"start": 521.2, "end": 528.48, "text": " the beginning. So what you'll basically be ending up in like bad states. And that will help this"}, {"start": 528.48, "end": 534.48, "text": " network become robust enough so that you can couple like cope with different deployment scenarios."}, {"start": 534.48, "end": 539.76, "text": " Because if we were feeding the perfect information coming from this, from these environment"}, {"start": 539.76, "end": 546.08, "text": " variables, then it will the basically the adaptation module would only be exposed to good"}, {"start": 546.08, "end": 551.6800000000001, "text": " trajectories. And that's not what we want, because we want it to be able to cope even with these bad"}, {"start": 551.6800000000001, "end": 557.6, "text": " trajectories when when something goes wrong. So that's the that's that's it in a nutshell. So in"}, {"start": 557.6, "end": 563.6800000000001, "text": " the first again, in the first part, they train the base policy. So that's your RL agent, they train"}, {"start": 563.68, "end": 568.3199999999999, "text": " this encoder of the environment factors. And they train this in the second stage, they train this"}, {"start": 568.3199999999999, "end": 574.7199999999999, "text": " adaptation module, which is really important, which learns how to predict the latent, so condensed"}, {"start": 574.7199999999999, "end": 580.8, "text": " information about the environment from the past observations only. And interesting thing to notice"}, {"start": 580.8, "end": 585.68, "text": " here is that so they have no vision component in the system. So all of the videos you clips, the"}, {"start": 585.68, "end": 594.4, "text": " clips you saw earlier were based solely on this RL and supervised learning policies. Okay. So"}, {"start": 594.4, "end": 600.0799999999999, "text": " deployment, what I do is they obviously feed the the past observations into the adaptation module,"}, {"start": 600.0799999999999, "end": 607.28, "text": " it runs on 10 10 hertz, because you have like 5050 past observations here. So it's kind of needs to"}, {"start": 607.28, "end": 614.0799999999999, "text": " run a bit slower. And it infers this this ZT hat, which is information about the environment,"}, {"start": 614.08, "end": 618.1600000000001, "text": " every now and then. So these two components are pretty much asynchronous. So that means that the"}, {"start": 618.1600000000001, "end": 624.08, "text": " base policy will be running on 100 hertz. So so it would be feeding the state and the last action,"}, {"start": 624.08, "end": 630.24, "text": " and it will just be fetching whatever the last update to this, to this vector of this latent"}, {"start": 630.24, "end": 637.2800000000001, "text": " vector is. So that assumption kind of is valid because environment won't be changing. Like"}, {"start": 637.2800000000001, "end": 641.6, "text": " empirically, they kind of figure out that it won't be changing, they don't need to improve the"}, {"start": 641.6, "end": 646.08, "text": " the frequency of this, and it just works. Okay, having explained that, let's now jump into some"}, {"start": 646.08, "end": 652.0, "text": " details. First thing, so this is, they notice here that the key, the key contribution of the"}, {"start": 652.0, "end": 657.44, "text": " paper is basically, so our novel aspects are the use of a varied terrain generator, and natural"}, {"start": 657.44, "end": 662.0, "text": " reward functions. So by the way, let me just digress here. So as you can see here, this is"}, {"start": 662.0, "end": 668.88, "text": " the terrain, it's totally random, it looks really hard, you can, the legs can get jammed, etc. So"}, {"start": 668.88, "end": 675.92, "text": " the whole point of this is, similarly to the OpenAI's robotic hand, is to train this thing in a bunch"}, {"start": 675.92, "end": 680.88, "text": " of different hard environments, I forgot to mention that, where they sample different parameters such"}, {"start": 680.88, "end": 687.52, "text": " as mass, and like in the OpenAI paper, they sample gravity even. And so the idea is to expose the"}, {"start": 687.52, "end": 692.72, "text": " policy to such a wide range of randomized worlds, so that when you deploy it in a real world, it"}, {"start": 692.72, "end": 698.08, "text": " just looks like another simulation, and that's the whole point. Simulation hypothesis confirmed."}, {"start": 698.08, "end": 707.0400000000001, "text": " So let's continue here. So the aspects are, as I said, varied terrain generator, and natural reward"}, {"start": 707.0400000000001, "end": 713.44, "text": " functions, motivated by bioenergetics, which allows us to learn walking policies without using any"}, {"start": 713.44, "end": 717.44, "text": " reference demonstrations. But the truly novel contribution of this paper is the adaptation"}, {"start": 717.44, "end": 723.2800000000001, "text": " module, trained in simulation, which makes the rapid motor adaptation possible. Nice."}, {"start": 723.28, "end": 728.88, "text": " Um, what next? So here is the reward function they're using. I mentioned it's heavily shaped,"}, {"start": 728.88, "end": 736.0, "text": " as you can see here. So the goal is the following. The goal is to, they say it's somewhere here, so"}, {"start": 736.0, "end": 739.6, "text": " the reward function encourages the agent to move forward with a maximum speed, blah blah blah,"}, {"start": 739.6, "end": 746.4, "text": " and penalizes it for jerky and inefficient motions. So basically you can see here, the goal is to"}, {"start": 746.4, "end": 752.24, "text": " maximize the reward, and if we plot this first component, so the whole reward will just be a sum"}, {"start": 752.24, "end": 758.88, "text": " of these 10 components, and if we just plot this thing here, so basically on the x-axis you have"}, {"start": 759.44, "end": 765.12, "text": " vx, and let me just kind of briefly digress here. So this is a robot, just to explain the coordinate"}, {"start": 765.12, "end": 771.28, "text": " system. This is a robot, pretty retarded, okay, but yeah, it works. And this would be the x-axis,"}, {"start": 771.92, "end": 778.4, "text": " y-axis would be, for example, pointing towards us, and these z-axis points upwards. And having said"}, {"start": 778.4, "end": 786.48, "text": " that, so this velocity x is just a component along this x-axis here, and so it basically increases"}, {"start": 786.48, "end": 793.04, "text": " until 0.35, so this value here, and then it just starts getting into the saturation, which means"}, {"start": 793.04, "end": 799.92, "text": " the robot is encouraged only to get to 0.35 and not above. So they, again, penalize, because this"}, {"start": 799.92, "end": 805.36, "text": " is minus, we want to minimize this part, so we want to minimize the sideways movement, so this is,"}, {"start": 805.36, "end": 810.64, "text": " again, y-axis coming out of the screen towards you, so we want to minimize that velocity component"}, {"start": 810.64, "end": 816.16, "text": " there, as well as this angular velocity, and like a bunch of similar stuff, like torque, we want to"}, {"start": 816.16, "end": 824.32, "text": " minimize torque, we want to minimize the joint motion, we want to minimize the slippage of the"}, {"start": 824.32, "end": 830.16, "text": " foot. So basically what this g of t is, they say it here, so blah blah blah, the binary foot contact"}, {"start": 830.16, "end": 836.8, "text": " indicator vector as g, so that's g. So that means you have four legs, and because you do this dyac"}, {"start": 837.36, "end": 842.24, "text": " operation, what you do is you basically get a matrix where you have a binary matrix, so you have"}, {"start": 842.24, "end": 849.92, "text": " maybe zero, one, zero, zero, one. That means two of the legs are currently in the contact with the"}, {"start": 849.92, "end": 857.04, "text": " floor, and only those, we'll only care about those velocities of the foot. So I think, so they say"}, {"start": 857.04, "end": 862.56, "text": " somewhere here, so vf is the velocity of the feet, okay? So that means we want to minimize that"}, {"start": 862.56, "end": 866.48, "text": " velocity, which makes sense, because that means if the velocity is hard, that means that the legs are"}, {"start": 866.48, "end": 871.04, "text": " kind of slipping around, and that's not something we want. We want a stable robot without jerking,"}, {"start": 871.04, "end": 879.12, "text": " without like these movements, like either sideways movements, or here you can see it's also"}, {"start": 879.12, "end": 884.88, "text": " penalized not to go up and down too much. So anyways, I can just imagine how it looked like"}, {"start": 884.88, "end": 892.96, "text": " to shape this thing. Not the best design, I guess, but like I love the results they got, and yeah."}, {"start": 893.92, "end": 900.24, "text": " In general, it's not recommended to shape the reward like this, but I guess their contribution"}, {"start": 900.24, "end": 906.08, "text": " lied somewhere else, and they, basically in the adaptation procedure, I guess this part is not"}, {"start": 906.08, "end": 911.36, "text": " as important as the fact that they show that their robots can adapt to unknown environments,"}, {"start": 911.36, "end": 917.52, "text": " even being trained in simulation only, that's a huge, huge thing. And they say it here again,"}, {"start": 918.4, "end": 925.12, "text": " this tells us about the fragility of this shaping. We found these reward functions to be critical"}, {"start": 925.12, "end": 930.08, "text": " for learning realistic gates in simulation, so I guess they had to iterate a lot to get this to"}, {"start": 930.08, "end": 936.32, "text": " work. And again, problems in the paradise, they say here, if we naively train our agent with the"}, {"start": 936.32, "end": 941.5200000000001, "text": " above reward function, it learns to stay in a place because of the penalty terms on the movement of"}, {"start": 941.5200000000001, "end": 947.84, "text": " the joints. So that translates into this. So this Q with the small dot here, if you can see, just"}, {"start": 947.84, "end": 954.72, "text": " denotes the velocities of the joints. So this term of the reward function will encourage the robot not"}, {"start": 954.72, "end": 962.1600000000001, "text": " to move at all, and yeah, as you can see here, if we just start with these, because this will be just"}, {"start": 962.16, "end": 966.56, "text": " a weighted sum of these rewards, right? And you'll have some parameters, as you can see here, they"}, {"start": 966.56, "end": 971.8399999999999, "text": " just figured out these parameters empirically. So if you just start with these parameters,"}, {"start": 971.8399999999999, "end": 976.9599999999999, "text": " the thing won't work, the robot won't move. So they do some sort of annealing, they start with"}, {"start": 976.9599999999999, "end": 982.0799999999999, "text": " super small coefficients, and then during the training, they slowly raise this up. So it's a"}, {"start": 982.0799999999999, "end": 988.0, "text": " training curriculum, obviously. And yeah, and they say to prevent the collapse, we start the training"}, {"start": 988.0, "end": 992.4, "text": " with very small penalty coefficients, so very small penalty coefficients, and then gradually increase"}, {"start": 992.4, "end": 996.72, "text": " the strength of these coefficients using a fixed curriculum. We also linearly increase the difficulty"}, {"start": 996.72, "end": 1001.44, "text": " of other perturbations such as mass, friction, and motor strength as the training progresses."}, {"start": 1001.44, "end": 1005.28, "text": " We don't have any curriculum on the trains and start the training with randomly sampling the"}, {"start": 1005.28, "end": 1011.2, "text": " train profiles from the same fixed difficulty. Bottom line, certain parts of the environment are"}, {"start": 1011.2, "end": 1016.72, "text": " under curriculum, certain parts like the terrain itself is always sampled from the same very same"}, {"start": 1016.72, "end": 1022.48, "text": " difficulty. That was the whole point. Here in this table, they just show the range they've used for"}, {"start": 1022.48, "end": 1029.04, "text": " certain parameters in the environment like friction, center of mass, etc. So yeah, so again,"}, {"start": 1029.04, "end": 1035.92, "text": " center of mass is this thing here, they have this robot, they put like an object with mass on top of"}, {"start": 1037.3600000000001, "end": 1044.08, "text": " its back, and they just test how it behaves under these external forces. That's the whole point."}, {"start": 1044.08, "end": 1049.4399999999998, "text": " And as I mentioned, this is the same thing as with OpenAI's robotic hand. They sample an environment"}, {"start": 1049.4399999999998, "end": 1054.3999999999999, "text": " where they'll be sampling uniformly from these ranges, and these ranges as well will be"}, {"start": 1054.3999999999999, "end": 1058.08, "text": " progressively becoming harder and harder as the training progresses, so that's because they are"}, {"start": 1058.08, "end": 1065.1999999999998, "text": " included in the training curriculum. One thing worth mentioning is that they mention it here."}, {"start": 1065.1999999999998, "end": 1071.28, "text": " Note that instead of predicting ET, so that's the environmental vector, which is the case in typical"}, {"start": 1071.28, "end": 1079.2, "text": " system identification, we directly estimate extrinsics ZT that only encodes how the behavior"}, {"start": 1079.2, "end": 1085.36, "text": " should change to correct for the given environment vector ET. So the whole point here is something"}, {"start": 1085.36, "end": 1091.44, "text": " similar to Mu0. If you watched that video, if you want to check it out, I'll link Mu0 video somewhere"}, {"start": 1091.44, "end": 1096.8, "text": " here, I made it a while ago, and the whole idea is the following. You only need to encode that"}, {"start": 1096.8, "end": 1103.04, "text": " information which makes you behave, have a better policy. You don't need to be able to reconstruct"}, {"start": 1103.04, "end": 1108.6399999999999, "text": " from that latent, so in this case, this environment vector, or in the case of Mu0, the observation"}, {"start": 1108.6399999999999, "end": 1114.8, "text": " space, you just need to have such information upon which you can act and have an optimal policy,"}, {"start": 1115.44, "end": 1122.96, "text": " and that's it. So you just save some bandwidth on modeling only the information you actually need"}, {"start": 1122.96, "end": 1129.28, "text": " to behave properly in the environment, and you just discard everything else. Okay, here we have"}, {"start": 1129.92, "end": 1133.8400000000001, "text": " some experiments they've done with certain baselines. So again, they have a bunch of"}, {"start": 1133.8400000000001, "end": 1139.44, "text": " different tasks. They have this uneven foam, they have this upward incline, so they're testing it"}, {"start": 1139.44, "end": 1146.0, "text": " in a bunch of different mattresses. They're like step up eight centimeters, step up six centimeters,"}, {"start": 1146.0, "end": 1152.32, "text": " step down 15 centimeters, et cetera. And they just compare it with the other baselines such as the"}, {"start": 1152.32, "end": 1158.4, "text": " this very same model, just without that adaptation block, as well as this A1. So A1 is just the name"}, {"start": 1158.4, "end": 1166.48, "text": " of the robot, so A1 is some default controller software that comes as a firmware of this robot"}, {"start": 1166.48, "end": 1171.68, "text": " when you buy it. And they showed that on all of these environments in these tables, just looking"}, {"start": 1171.68, "end": 1177.92, "text": " at the success metric, TTF, so it's time to fall and distance, RMA pretty much outperforms the other"}, {"start": 1177.92, "end": 1183.1200000000001, "text": " baselines in all of these tasks, as well as in this payload analysis, we have these diagrams"}, {"start": 1183.1200000000001, "end": 1189.44, "text": " where we are on the x-axis, we are increasing the weight of this thing of the payload on the robot's"}, {"start": 1189.44, "end": 1194.0, "text": " back, and we can see how different baselines behave, and we can see that the success drops"}, {"start": 1194.0, "end": 1200.4, "text": " really slowly with this RMA approach, whereas with the baselines, it drops pretty significantly."}, {"start": 1200.4, "end": 1206.48, "text": " And worth mentioning is that the robot itself has 12 kilos, which means that at this point on x-axis,"}, {"start": 1206.48, "end": 1212.3200000000002, "text": " where basically the robot is carrying the same amount of weight as its own body weight on its"}, {"start": 1212.3200000000002, "end": 1220.8000000000002, "text": " own back. So that's pretty cool. So here they have another analysis on this particular environment"}, {"start": 1220.8000000000002, "end": 1227.92, "text": " with the mattress covered with oil, and what you can notice here is that from this point on,"}, {"start": 1227.92, "end": 1235.8400000000001, "text": " so when it starts stepping on this slippery floor, the environment parameters obviously change,"}, {"start": 1235.8400000000001, "end": 1241.04, "text": " like the friction parameter will change, and we can see that being"}, {"start": 1241.8400000000001, "end": 1248.4, "text": " inferred by the adaptation module exactly here. So we just log, so this latent vector, I think"}, {"start": 1248.4, "end": 1254.88, "text": " it has like eight dimensions, so they just took Z1 and Z5, so those are two particular features"}, {"start": 1254.88, "end": 1261.2800000000002, "text": " of that latent vector, and they just do, they plot it, and we can see that it basically reacts"}, {"start": 1261.2800000000002, "end": 1266.8000000000002, "text": " really quickly to the change of the environment, and that in turns helps the robot. So you can see"}, {"start": 1266.8000000000002, "end": 1275.92, "text": " here, this is the gate patterns. Basically RL means rear left leg, rear right leg,"}, {"start": 1276.64, "end": 1282.8000000000002, "text": " like forward left leg, and forward right leg, and so you can just see these means that"}, {"start": 1282.8, "end": 1288.48, "text": " so when you have a bar here, that means that the contact is made between this particular leg and"}, {"start": 1288.48, "end": 1294.24, "text": " the floor, and you can see that things start getting messy here at one point in time, and then"}, {"start": 1294.24, "end": 1299.6, "text": " the pattern returns to normal even though the robot is still on the environment because of the"}, {"start": 1299.6, "end": 1306.1599999999999, "text": " adaptation module, and also here what I mentioned is that the peaks here, and this is the torque"}, {"start": 1306.16, "end": 1313.8400000000001, "text": " plot as you can see here, the knee torque gets higher after this part here, and that's just"}, {"start": 1313.8400000000001, "end": 1321.76, "text": " a way to visualize what happens internally in that RL agent and adaptation modules, and yeah,"}, {"start": 1321.76, "end": 1330.0800000000002, "text": " pretty awesome behavior. Some final results here are in simulation. They just show a couple more"}, {"start": 1330.08, "end": 1335.9199999999998, "text": " baselines. We have RMA, we have RMA without the adaptation module, we have expert which is using,"}, {"start": 1335.9199999999998, "end": 1340.96, "text": " instead of using the previous observations to figure out the latent environment vector,"}, {"start": 1340.96, "end": 1346.24, "text": " it just uses the ground truth vectors, and we can see looking at all these metrics like reward,"}, {"start": 1346.24, "end": 1353.1999999999998, "text": " TTF, so time to fall, distance, samples, torque, smoothness, if you remember stuff like torque"}, {"start": 1353.2, "end": 1360.4, "text": " and stuff like smoothness, and ground impact were like enforced to be minimized by the reward"}, {"start": 1360.4, "end": 1366.16, "text": " components up there, so let me just, for example, let's take a ground impact or smoothness. If I go"}, {"start": 1366.16, "end": 1372.72, "text": " there to the reward function, so here we have, so smoothness as you can see here, or ground impact,"}, {"start": 1372.72, "end": 1380.16, "text": " that means that the force, so F is the, so ground reaction force is at the feet as F,"}, {"start": 1380.16, "end": 1385.1200000000001, "text": " so this part just tells you how elegantly, let's call it that way, the robot is kind of touching"}, {"start": 1385.1200000000001, "end": 1391.1200000000001, "text": " the floor, and it should be minimized looking at this because, yeah, because it's a part of the"}, {"start": 1391.1200000000001, "end": 1397.44, "text": " component with a minus prefix here, and just again returning to the table, we can see that the"}, {"start": 1398.48, "end": 1404.0, "text": " ground impact gets smaller and smaller here, and we can see that the difference between RMA and"}, {"start": 1404.0, "end": 1409.76, "text": " expert is really small at all of these metrics, so we can see that the ground force is at the"}, {"start": 1409.76, "end": 1416.08, "text": " feet, so just comparing the success rate, comparing the TTF, comparing reward, comparing distance,"}, {"start": 1416.08, "end": 1424.32, "text": " and everything else, it seems that RMA is able to infer the latent environment vector almost as"}, {"start": 1424.32, "end": 1429.76, "text": " well as the ground truth vector, so that's awesome, that means that the supervised learning"}, {"start": 1430.72, "end": 1435.52, "text": " like knowledge distillation worked, so that's pretty much it, that was a pretty neat idea,"}, {"start": 1435.52, "end": 1443.84, "text": " awesome results, this has, this can have amazing implications on RL being used in real world"}, {"start": 1443.84, "end": 1448.8, "text": " environments combined with some other types of learning, paradigms such as supervised learning,"}, {"start": 1449.68, "end": 1455.52, "text": " and like I guess the like follow-up work to this thing will be integrating vision component and"}, {"start": 1455.52, "end": 1459.52, "text": " then just deploying it, and next thing you know, Boston Dynamics robots will be running around the"}, {"start": 1459.52, "end": 1464.48, "text": " city, that's pretty much it, hopefully you like this video, if you did, share it out, comment down"}, {"start": 1464.48, "end": 1470.16, "text": " below if you have any suggestions for the next papers or whatever you have to say, and until next"}, {"start": 1470.16, "end": 1498.3200000000002, "text": " time, bye bye!"}]