OpenAI - Solving Rubik's Cube with a Robot Hand | RL paper explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I cover OpenAI's famous robotic hand that learned to solve the Rubik's cube by training purely in a simulation.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/1910.07113
✅ Blog: https://openai.com/blog/solving-rubiks-cube/
✅ Quaternions: https://www.linkedin.com/posts/aleksagordic_visualizing-quaternions-an-explorable-video-activity-6779354558696087552-0heH
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
01:32 Comparison with Dactyl system
03:01 High-level overview
07:00 Tasks (Rubik's cube and block reorientation)
08:55 Physical system overview
11:08 Reading angles from the cube (electronics)
13:25 Realistic modeling of the system in simulation
15:52 Automatic Domain Randomization (ADR)
19:50 Cube size randomization during training (blog)
20:30 Entropy and rand param probability distribution
23:55 ADR pseudocode
27:15 Rapid
28:45 Randomizations
31:35 PPO
32:05 Actions and rewards
33:55 Policy network, embed and add
37:35 Behavioural cloning
39:55 Vision pipeline
41:10 Focal loss
42:10 Results
47:30 Perturbation robustness
48:20 Meta-learning 
51:35 Predicting environment variables from LSTM hidden state

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#robothand #openai #rubikscube
 What's up? In this video I'm gonna cover this paper called solving Rubik's Cube with a robot hand by the pretty much complete OpenAI team as you can see here and the goal here is to solve the Rubik's Cube using a robotic hand obviously and but the thing I want you to understand is that the point is not to actually figure out how to solve the cube they just use some off-the-shelf solver for that the whole the difficulty in the project lies in sensing the state of the hand the state of the cube and advising a control policy I controlling those joints on the hand in order to solve the cube so it's a control and sensing problem not the actual how to solve the cube problem and having said that take a look at this video we're trying to build robots that learn a little bit like humans do by trial and error what we've done is train an algorithm to solve the Rubik's Cube one handed with a robotic hand which is actually pretty hard even for a human to do we don't tell it how the hand needs to move that the cube in order to get there the particular friction that's on the fingers how easy it is to turn the faces on the cube what the gravity what the weight of the cube is all of these things it needs to learn by itself awesome hopefully you got some high-level understanding from that video and now let's just dig into the paper it's a quite a long paper hopefully you'll at the end of this video I'll understand it much better okay let's start with this the two main components that this new paper advised because as a side note they previously published this paper called learning gestures in hand manipulation where the setup was pretty similar the only difference is that they are trying to just reorient this block that has these open AI letters on every single face of this cube and so the goal was much simpler you just need to flip the cube and to get from a certain source position into a certain target configuration and that's it it's still a complicated task but solving the Rubik's Cube is just a level up so yeah and the two main components they say are key to solve this problem the Rubik's Cube is this automatic domain randomization algorithm and a new robot platform built from machine learning and high level this automatic domain randomization so they in the previous paper and this one they already used domain randomization but the novelty is this in this automatic part so they are automatically learning how to create progressively harder and harder environments and by doing that it turns out that the policy once deployed on a real robotic hand treats the like the real world as just another simulation matrix confirmed and let's dig into the high level explanation so as you can see here the the setup looks something like this so they have bunch of these simulations and you can see every single hand corresponds to different simulation and everything is randomized from the appearance you can see different the color hands the physics like the gravity the cube size the cube mass the friction of the robotic hand bunch of stuff is randomized so once you have the data once you have those simulations what they do is they train two separate pipelines so one is for training the control policy and you can see it here so it gets as an input it gets the fingertip position so you get pretty much five 3d points and it also gets the state of the cube so that's the position the orientation and the six face angles as we'll soon see and once you have that state of both the hand and the cube they train this LSTM based policy network in order to output these actions and that will be just 20 times 11 basically 20 means 20 joints so the hand has 20 joints they can control and 11 is just discretized angles because they later show that that's much better and the place is much better once they do that instead of using continuous actions so those are just the relative angles so maybe move this specific joint by two degrees and that's it on the other hand the second pipeline takes these three RGB images as the input and it tries to regress the the cube pose the cube orientation as well as the face angles and we'll see all of the details a bit later once they combine all of that together we have this one this this chart and you can see here that the cube pose always comes from the visual system but because it was really tough actually to figure out the absolute face angles so you have the cube and you have to figure out which exact angle and you have from minus pi to pi the angle goes from minus pi to pi so it's really hard to regress it because all for the occlusions etc and they sometimes actually use the cube itself it's a smart cube that has sensors and we'll see that in a moment and it basically just outputs and gives to the policy these numbers which represent the the face angles so they either use vision or they use the cube itself in order to get the face angles so combine them with the fingertip positions that's the input observation to the policy and they basically train the LSTM based policy to output the correct actions in order to solve the cube so that's the high level explanation now let's start digging into details they say here the the learning requires vast amount of training data which is hard and expensive to acquire on a physical system I should have mentioned this a bit earlier but basically you have two options you can either train this thing on a robot but because we're trying to learn we know we'll need a lot of data and it's simply not practical to train the policy on a physical robot and in the previous paper the system was called Dactyl just FYI basically they showed that the hand had a lot of breakages so it's super impractical to train this even if you have a farm of robots and this thing this this hand is pretty expensive I can assume so it's really hard to have a setup to collect enough data to train the policy so they have to go to simulation but once you're in a simulation you need to advise a good way to bridge the domain gap the reality gap to in order to have the policy training in sim be good enough in the actual real world so that's just something to keep in mind first of all let me just briefly explain so the goals the first goal we saw was the block reorientation so that was in there in the previous paper they were solving this one and they're using it as a baseline in this one so the goal is considered solved if the orientation of the cube falls inside this 0.4 radians from the target orientation and that's something like 22 degrees more or less 22 23 degrees so it's a pretty big margin but yeah they just figure out that number works for them okay now you can see here visualized so this is the the block reorientation task you can see the target orientation here and this is the Rubik's Cube task and you can see the target is the solved cube in this case they found that one interesting thing to notice here is that it's really hard to rotate certain once you have a single hand it's really hard to rotate any face other than the top face and they say here we found rotating the top face to be far simpler than rotating other faces so they said it's really hard to rotate this one if you have only a single hand so it's much easier to rotate the top face and thus instead of rotating arbitrary faces we combine together a flip and the top face rotation in order to perform the desired operation that these sub goals can then be performed sequentially to eventually solve the Rubik's Cube so just using those two operations the flip and the face rotation the top face rotation they can solve the cube when it comes to solving the Rubik's Cube competing a solution sequence can easily be done with existing software libraries like a Siam Bo solver so that's what I mentioned in the beginning of the video they use off-the-shelf solver and in this work the key problem is thus in sensing and control not finding the solution sequence so keep that in mind okay briefly just let me just go through this physical setup so they have the cage they have these three cameras one two three three RGB cameras they have this system called phase space which they use in order to track the fingertip positions and you can see these small LEDs on the fingertips and that's what a what this phase space system motion motion capture system uses in order to figure out the exact 3d positions of those fingertips so then we have the hand and we have the like the cage that holds everything together so one thing I actually when I first saw this video a couple of years ago I thought that the hand was also developed by open AI I wasn't sure which part of the project they actually developed but actually the the hand itself is from some other company it's called the shadow robot company and it's a pretty marvelous piece of engineering itself the hand itself I think has 24 degrees of freedom so it's a really like high dimensional control problem so previously nobody could solve nobody could imagine solving like rubik's cube using like a hand as complex as this one but some other systems exist but they are not learned and they are not like a humanoid hand they are just like some kind of a machine that can solve the cube really fast but that's not what we are trying to accomplish here aside from the hand and the setup there is the cube component and I already mentioned it's there is smart cube involved and they say here we therefore replace some of the components of the original Geekr cube with custom ones in order to achieve a tracking accuracy of approximately five degrees so the original cube only can so we need as I said we need to to calculate these angles and we need some better resolution than just nine degrees so we need to have resolution that's and they they figure out the five degrees works so they had to create a custom cube and that's funny and you can see here and I just highlight the charging here because it's not a simple it's not like dummy let's go that way a cube it's a it's got like a microcontroller it's got a couple of components in order to figure out those angles and we'll see that in a moment okay a quick detour into how this exactly works is you have this thing in every single face of the cube and it's basically a variable resistor so here is just circuit on the left side and you can see this variable resistor here actually corresponds to this thing here and this is just a simple circuit you've got resistors we've got the positive power rail we've got the ground ie the zero potential in the circuit you have something called operational amplifier here and what this circuit does you can treat it as a black box and most people do basically just hold the same potentials on the plus and on the negative inputs and all in all you've got a current going through here and the signal here is basically our times I where I is the current and our is changing so the analog signal will actually be increasing as the resistance increases and the resistance increases once you're turning the the face particular face and so what I do is they have a microcontroller here inside of the cube so some microcontroller it's got something called analog to digital converter and basically this signal here is being fed into a DC and it can discretize the signal into some value so maybe let's say this range goes from from 0 to 5 volts when you change the when you rotate the cube from 0 to 360 degrees and you can just discretize that into maybe some of the numbers will be you'll just have some kind of a curve and this is the analog analog voltage and this will be the output from the converter and you can then map those numbers into angles and send them via Bluetooth to your policy and that's the part of the observation that the policy uses in order to control the joints and eventually solve the sub goals and eventually solve the cube so that was a quick detour into electronics it's funny to see a piece of electronics inside of enamel paper actually studied electronics it's pretty satisfying feeling to to see that you can actually leverage some of the knowledge to learn on the faculty yeah this will probably be the first and the last time I'll ever see it anyways let's continue and let me tell you one more thing before we start jumping into how the policy is trained and etc so you have the setup but you want to have the simulation to be as as closely to you want your simulation to resemble the real world as closely as possible so they showed here that they added additional modeling to the hand like these pulleys and tendons and doing that they improved the simulated hand and you can see on these charts basically they track one particular joint like maybe this one and they track the its angle and you can see the this curve the green one is you get these curves by taking a control sequence and sending that control sequence both to the robotic hand to the real robotic hand and to the simulation and you're just tracking the 3d the angle of that particular joint and you can see there is a discrepancy before the calibration before this modeling was done once they included that you can see that the curves align which means we have much better simulation and thus we'll have better control policy once we deploy it on a real robot and they show this is really important part of their of their paper okay so let me jump to the cube so it turns out modeling the cube is super hard there is so many details like the friction the size the mass the lots of forces inside the cube itself so aside from the forces that act inside of the cube there's this thing called a forgiveness factor so once you have the top face and I rotated maybe by a couple of degrees I can still take this perpendicular face and rotated and it will snap and it will work but like if you if you go too far now you can't you basically can't flip the the perpendicular face so that's the forgiveness factor so you have to count all of those tiny details inside your model in order to get a realistic representation so this here they performed a very rudimentary dynamics calibration of the parameters so like the like the size the friction etc which mojoco which is the engine they're using for rendering allows us to specify in order to roughly match a physical Rubik's Cube our goal was not to get an exact match but rather to have a plausible model as a starting point for domain randomization so again ADR will see that that plays a really important part in this work okay and here we are so and they say here so in this previous paper so that's the deck tile the the previous paper they published we were able to train a control policy and a vision model in simulation and then transfer both to a real robot through the use of domain randomization however this required a significant amount of manual tuning and a tight iteration loop between randomization design in simulation and validation on a robot in this section we describe how automatic domain randomization can be used to automate this process so it's a mouthful basically what it means is a following so in previous paper they for every single parameter that they want to randomize like maybe the cube size they had to manually find the range so they had to find maybe this is a 5.4 centimeters this may be 5.9 and once they said those basically margins you can sample anything in between but you can't sample outside that range so that means you have to for every single range you have to test whether that actually leads to a better policy when you deploy it into a robot because if you make this arbitrary big maybe this goes to 7.2 now the policy will actually be much worse because it's really hard for the policy to learn how to solve some of those cubes and then it leads to a worse policy which is not desirable so what the ADR basically does is it starts from a single value so they measure the realistic cube and that was 5.7 actually in this case and then they slowly expand to the left and to the right and they end up with some a bit bigger range and then a bit bigger range and so that's kind of an implicit curriculum learning and we'll see those details so that's the whole point so for every single randomization parameter like maybe the like the friction the cube mass the cube size even the gravity etc they gradually and automatically kind of increased the actual range for that parameter and that's the whole idea the high-level explanation of ADR they say it here the main hypothesis that motivates ADR is the training on a maximally diverse distribution of environments leads to transfer via emergent metal learning more concretely if the model has some form of memory and they use LSTMs and we'll see that in a moment it can learn to adjust its behavior during deployment to improve performance on the current environment over time I by implementing a learning algorithm internally so basically what happens is that you have a finite capacity policy and you've got pretty much infinite number of sampled environments so that policy can't can't memorize every single solution to every single environment and there is also not a single robust policy that can solve each of these really in a performant way so it has to use the memory it has in order to adapt to the given environment and improve its skills that way and they later show that actually that happens and the policy learns how to encode the environment parameters like cube size cube mass gravity etc inside the hidden state of an LSTM and that will be an important thing they'll want to show you okay so here is a high-level explanation and I already gave you here so they are basically just updating those ranges and they say here at its core ADR realizes a training curriculum that gradually expands the distribution of environments for which the model can perform well the initial distribution of environments is concentrated on a single environment for example in policy training the initial environment is based on calibration values measured from the physical robot so they measure for example the cube so it's 5.7 centimeters the gravity is like 9.81 whatever so they have these initial values and then they gradually expand them and I'll show you in a blog in a second how that looks like here is a really nice visualization on their blog you can see that the x-axis is the training the timeline basically and the y-axis is the the range I was mentioning so they start with 5.7 centimeters but but as the time progresses the ADR does its job and the cube eventually comes to this huge range so they actually sample some of the cubes will have 5.47 centimeters size and some will have 6.13 so that makes it much harder to solve the tasks but because they gradually went to that point the policy learns how to cope with those cube size ranges getting back to the paper let me explain the pseudocode for this algorithm but previous prior to that let me just explain this briefly these two equations so they've got this distribution here and the entropy so as what they did is for every single parameter that they want to randomize like like the friction and gravity they'll have a uniform distribution so for the cube size for example 5.5 I don't know like maybe 5.9 they'll just have those ranges and they'll have a uniform distribution here so and that they'll have the same thing for every single parameter so this is maybe like cube size blah blah blah and they'll have like friction etc so the final distribution will just be a product 1 to D where D is the number of those parameters and they'll just multiply all of these uniform distributions fi low means the the lower the lower part of the range so that's 5.5 for the cube and the high part means the high threshold and that's it and once you have this distribution you can find its entropy and for uniform distributions it's quite easy so the entropy is basically in general's case for a discrete entropy for discrete distribution you'll have a simple sum of probability of that event times the logarithm of 1 over the probability of the event and this part the log part is actually the self-information so it's sometimes denoted like I of X and this is how you calculate the entropy and basically because the uniform distribution always has the same values so maybe if we take a simple discrete distribution like this one let's say we have three possible events so all of them will have 1 over 3 probability and so you can see so the probability is 1 over 3 and then you have the log 1 over 1 over 3 so that's 3 just 3 and we have that 3 times so these will just cancel out and you end up with log 3 so in general case this is just the difference they denote here so in the continuous case it will just be your entropy will be the integral pi of X log 1 over pi of X DX and because for a particular uniform distribution this is constant you can just pull it out here and so we've got integral over DX and that will just be the size of your domain so maybe here it will be whatever the difference between these two are and that's why they have the difference here so this thing is just the entropy of a uniform distribution that was a long way to say that thing and then they just divide by the number of distributions by the number of randomized parameters and you get kind of a average entropy and so the the the more spread out these distributions are the higher this entropy will be because you can see as the as the range increases the entropy increases and so this is a measure of how actually demanding the environment the current environment is so that's just a simple way to do that okay here is the actual algorithm they start with some initial calibration values as I mentioned that may be 5.7 centimeters for the cube 9.81 for the gravity and they keep these buffers for every single one randomized parameter and they keep the lower and the higher so one for the lower range part of range and one for the higher part of the range and they keep these these are just thresholds for how performant the policy is given a specific environment and M just means so M is just a number of iterations we need in order to decide whether we want to expand or squish the range of that particular randomized parameter okay so we sample a particular vector from the current distribution and we pick random randomized parameters so one of those D parameters that we are using we pick one of them we then sample from X from the uniform distribution and with 50% chance will either pick the lower so we'll set lambda I to the lower extreme or with 50% chance will set lambda I to a higher extreme so what that means in that particular case with the Rubik's Cube that means that sometimes so basically once you sample this you'll sometimes take your sample 5.6 blah blah but like in this case if you pick the lower one you'll set that lambda I that corresponds to the Rubik Cube size to 5.5 or to 5.9 so that's this part and basically then you evaluate the performance of the policy on that specific environment so hopefully understand what happens here so you pick all of the parameters or all of those the randomized parameters you have randomly from their ranges but you pick one specific one and you intentionally pull it to some extreme either to the lower extreme or to the higher extreme and you evaluate the performance you save the performance inside of the buffer and you keep doing that until you see this threshold M so that may be 128 times once you have enough samples you're you're confident that your estimation will be good and you just take the average performance you just this is just clearing the buffer and if the average performance is good enough whatever this TH is depending on whether you're training vision vision network or the policy network so if you're good enough that means that the network that was that is being trained doesn't have any difficulty solving this particular environment so we can expand the range and next time the environment will be even harder on the other hand on the on the flip side basically if the performance is really poor that means that pulling this particular parameter to its extreme is really hard and we want to actually squish this this uniform distribution for that particular randomized parameter so hopefully this was clear enough let me know in the comments because this is arguably the most important algorithmic innovation they had in this paper other than that there was a lot of engineering and nothing less important but yeah okay let's continue here is just a high level overview of how the vision pipeline and how the policy pipelines are trained you have the model parameters you have the current PHYs so those are the the the ranges we've just mentioned for every single randomized parameter and you you basically sometimes sample those you generate the data and you pull push the data into this buffer but we sometimes just do the ADR so basically snap one of those parameters to its extreme and we sometimes update this PHY to make it more challenging or to reduce its difficulty and that's the two things are happening parallel we are constantly improving the the difficulty of environment and we are constantly generating data and training the model and updating the model parameters so yeah that's how it works and you can see one one difference between how they train the vision network compared to the policy is that they here they use both the data that is evaluated as well as the actual data pulled from the current parameters and they do that because data is really expensive and they need all data I can have for that particular policy the problem with with this thing is that sometimes you'll be pulling data that's too hard because this is still remember this is still under evaluation we still don't know whether snapping that parameter to its extreme is too hard or not and then yeah there's just the trade-off but they figure out it's good enough for their approach okay let's now see what they randomize in their simulations and already mentioned like the physics like geometry friction gravity etc the interesting part is this one the adversarial what they do here is the following so they have their training the policy network that's the P and it's outputting those 20 times 11 categorical distributions that correspond to angles for particular joints and on the other hand they have this network the adversary called are which just basically outputs random actions and those get added to these ones and the policy needs to learn how to complete the sub goal it's given and disregard the randomness introduced by the adversary and they tried two things actually try to intentionally to develop a true adversary which will try and perform the worst actions in order to kind of confuse the policy but they realize that just using a random network is actually even better because it brings more diversity and later the policy can better generalize once deployed on a real robot so why do they do this and the simple answer is the following basically once you have the real hand even though the policy says to a specific joint move by maybe two degrees the underlying controllers like the PID controller will add some noise so it will maybe execute 2.1 degrees instead of 2 degrees and then the policy will still have to learn how to deal with that so and that's why they introduced this kind of modeling into the pipeline they also add noise to the observations basically fingertip positions are kind of added like they had a Gaussian noise to to the positions etc so everything you can imagine everything that's in this pipeline they try and randomize finally the vision like the lighting conditions was the lux level the camera positions so remember they have this cage their cage and they figure out the positions and use those as the calibration as the initial as initial positions and then in the simulation they'll just have some volume that will be expanding and they'll be sampling camera positions from that volume and that will introduce randomness they'll also experiment with the field of view with bunch of different parameters and yeah that's pretty much it and I mentioned here so some of these are fixed in a specific simulation so they'll just pick the cube size and the mass and they'll freeze that but other parameters like the observation noise and adversaries like the the thing I just mentioned will be changing with every single step in the episode so yeah just keep that in mind and finally training the policy they use this thing called PPO proximal policy optimization which is basically actor critic method they devised a couple of years ago it's a fairly standard RL algorithm I won't dig into too much depth here basically they have the value network they also have the policy network the value network tries to predict the expected value and the policy network tries to predict the optimal actions and in that collaboration they eventually learn a really good policy let me just focus briefly on on on these details like the rewards and actions so I already explained how the actions function so basically for every single joint and there are 20 of them they output these categorical distributions so maybe something like this and the x-axis is like the angle so here this is the highest probability angle so that maybe correspond to five degrees so they'll pick five degrees so those are the actions the rewards they have three types of rewards and they had some reward shaping here as you can see so the first component is the non sparse component and for example if they are trying to in the case of cube flipping if they're trying to get to the target orientation the reward will be the closer you get to the orientation the final orientation the more reward you'll have for each step so if you decrease the distance between those two orientations you'll be getting more reward and how they represent the orientations is using something called quaternions and those are just it's just an extension of the complex number system into four dimensions it sounds pretty complex I recently created a post on LinkedIn and I'll link it down in the description where I explained the best resources you can get to learn about quaternions on the high level you can treat them as a black box it's just a four-tuple basically four numbers and they somehow captured the notion of the orientation of the object and they are used all over the place in computer graphics and robotics etc so the second reward is a reward of five whenever the goal is achieved so that's a sparse reward as you can as you can assume and finally the penalty of minus 20 when the cube or the block is dropped so that's the reward system they they deploy here is the actual architecture you can see the as I mentioned the value the value network and the policy network so that's the part of the PPO algorithm and you can see the distribution 20 by 11 and value has just a single scalar and one thing interesting here is that the the the policy network takes the only the noisy observations whereas the value network takes both the non noisy observations which are only available in the simulation but also it uses the noisy observations and the reason we can use that all of the data that's not available on the actual robot is because we'll won't be using the value network at all we'll be just using the policy network so this is this a symmetric act or critic approach which they use and which they devised in some of their previous papers okay here are some of the inputs you can see like cube orientation quaternions fingertip positions blah blah blah you can notice that the only the noisy parts go into the policy networks you can see here the check sign so the noisy fingertip positions you can see the check sign here the Q position and the orientation etc okay a couple of interesting things to notice is this and these enable them to experiment a lot more is this embed and add approach so you can see here that the observations are always normalized and then embedded into this constant 512 dimensional dimensional space and they just summed it up summed it up and they apply value and what that enabled them to do is experiment with different observations and keep the experience they accumulated over over many many months intact and they say here so more concretely we're first embed each type of observation separately without any weight sharing into latent space of dimensionality 512 we then combine all inputs by adding the latent representation of each and applying a value non-linearity after the main motivation behind this change was to easily add new observation to an existing policy and to share embeddings between value and policy network for inputs that feed into both so for example if they have fingertips without noise here and fingertips with the noise they'll be sharing these embeddings in those cases otherwise they are not sharing the the weights there okay let's continue this is super interesting and mind-blowing so they actually train this thing they say here we've been training the Rubik's Cube policy continuously for several months at this scale while concurrently improving the simulation fidelity the ADR algorithm they've been tuning the hyperparams and they've been changing the network architecture so the cumulative amount of experience over that period used for training on the Rubik's Cube is roughly 13,000 years and which is on the same order of magnitude as the 40,000 years they used to train the open AI 5 the bots they train for Dota 2 game and basically what I want you to have in mind here is that a single simulation step takes is one is 80 milliseconds so you just divide this number here the number of milliseconds here by this and you'll have the number of steps that happen in the simulation since they started doing this and it's a huge number okay experimenting is really important obviously they want to keep the experience that they want to ditch the experience they accumulated and so they do one more thing and they say here so we therefore rarely train experiments from scratch because it's expensive yeah but instead updated existing experiments and initialized from previous checkpoints for both the ADR so they both keep those those thresholds for each of those ranges for the randomized parameters and they also keep the policy parameters and they did something called behavior cloning in order not to lose experience they implemented something called behavior cloning if you're familiar with imitation learning this is one of the basic approaches there and they just implemented a variant of this dagger algorithm and why they do this is that they can iterate with different policy architectures not only the observations and still keep the experience intact pretty much intact so what how it works is basically you have something called a student network you have the teacher network and this one is basically we input the observation out goes the actions and we use these actions to navigate the environment but what we additionally do is because it's got the value function here as well so these are the actions and here is a teacher so that's a that's the the best network we currently have with a slightly different architecture and this is the new network the new architecture and we want to kind of transfer the knowledge experience to that new network and so what we do is additionally we have the values here and we have the actions here and we just do MSC loss between these two so just simple mean square error loss there and we do simple KL divergence between those two and that will make sure that the student eventually learns to output the same value functions and to output the same actions in particular states as the teacher network and that's what what I mean by behavior cloning in this case okay so they say here this is work surprisingly well allowing us to iterate on the policy architecture quickly without losing the accumulated training progress our cloning approach works with arbitrary policy architecture changes as long as the action space remains unchanged so the only thing that needs to remain constant is the action space itself so 20 times 11 everything else can change the observations can change and the architecture itself can change so I think already mentioned estimating the face angles is super hard so what I did is they so they just added these cutouts so that the problem of estimating the angle is a bit easier and they say here so while predicting position and orientation works directly works well we found predicting all six face angles directly to be much more challenging did you have the occlusion even when using a cube with a symmetric center stickers to work around this we decompose face angle prediction into several distinct predictions so for the sake of brevity I'll just skip over the details but they estimate this thing called active axis and the active face angles and the top face angle in order to and then they do some post processing in order to use as a proxy to get the actual face angles and yeah that's what I do with the vision part and you can see the network here so position orientation which is a quaternion and these three elements combined with some post-processing indirectly gives them the face angle estimations one thing you'll notice is there is many components here and you kind of usually have to manually specify that when you do the weighted loss you have to manually specify those weights so a way around that is using something called focal loss and there is this paper which introduced it previously so then the main idea is the following so let's say you're doing classification and the true label is maybe one and so you're using cross entropy loss your loss will look something like this and let's say you output 0.9 and that means the loss is pretty small and what they do is the following they just wait using this this here this factor here so because the and the minus log P is the cross entropy part so this is the the actual weight and so you can see the bigger the P is so the closer we are to the correct solution the smaller the weight for this loss so that means roughly that if we are performing really good here we want ditch we want to drop the weight for that component because we are already really good there but if we're maybe struggling here we'll increase the weight that goes with this loss component and that's pretty much it and then they do the ablation studies and they show that using the full model that means using the focal loss that the domain randomization etc they achieve the best results looking at maybe a real cube you can see that the orientation error is seven point something you can see the other values and that's much better than when they don't use the randomization or no focal loss etc so yeah you can see the order of magnitude of these errors and it's not small but it's good enough let's let's continue and jump into results it shows some results on the sim to sim transfer performance and basically the model was the policy was trained in the ADR setting and the test set also comes from the simulation but using some manually specified ranges and so that's the the holdout data and they show that as the randomization level increases and that's the entropy I mentioned previously as it increases and it's a good measure of how random how big those ranges are you can see that the performance on the the number of successes on the of that on that holdout set gets better and that's one thing and the second thing is they show here the performance on the real data and using the ADR so the more randomization they add so you can see the entropy increases basically the better the results get if we take this specific policy the XXL policy which was trained for a lot of time you can see they even didn't they didn't put in the numbers here so you can assume a couple of months and you can see the the mean and the median number of successes in the block rotation problem increases and it's pretty good in the real world on the real robot as well okay let's continue and see what else is there this demonstrates that the curriculum learning is really important so the blue line is trained the blue policy is trained in the ADR setting whereas all of the other curves are manual using the manually set domain randomization ranges so what they basically do is the following so when you start with ADR we start with a single value and then you slowly expand the range and you expand the range again and they expand the range again and this may be like the cube size this is the size and how they produce these other curves is for example this one corresponds to maybe this range and the the worst one actually corresponds to the biggest range so what happens there is that if you start training the network with a huge range it's actually going to underperform because it didn't have the the benefit the the luxury of doing it gradually as it was just thrown into the water so to speak and it was really hard for the for that policy to cope so basically yeah this is a argument for using the curriculum learning pretty much here again they showed that on the block orientation problem as we increase the ADR ranges the entropy increases and the errors for estimating the orientation and position of the vision pipeline gets better and better you can see this gets much better as we increase the entropy similarly for the rub Rubik's Cube the orientation position and the top angle all increase going to higher entropy ADRs and that's that makes sense okay this is the final result they got on the Rubik's Cube basically what I did is they took the maximally hard scramble that means if you know some Rubik's Cube theory at the hardest state you can be in is 26 quarter rotations away from the like completed cube state that means quarter rotation is just a 90 degree rotation on the cube and it can be shown that there is only three states that there are 26 quarter rotations away from the goal from the completed cube state and they took one of those hard states and they use those to test the policy and they can show here they showed here that on the full scramble so and you basically have to have 43 successes to get to the completed state because we're only using remember we're just using the phase rotations and we're just using the flipping and basically we need 43 successes in order to successfully solve that particular scramble and they show that for the XXL ADR so highly randomized ADR and using the Giker cube for the face angles instead of vision because you remember it's hard to estimate those using vision and they show that they have 60% success rate on the half scramble and 20% success rate on the full scramble once deployed on a real robot and that's pretty amazing still not perfect but like it's pretty awesome yeah that's that result and let me continue and show you some more awesome results and you can see that using a rubber glove which changes the like effectively changes the friction of the environment or using the tide fingers or maybe putting the blanket over the cube which will basically render the vision pipeline useless because I can't figure out the position and orientation or they do some kind of funny perturbations with giraffes and pencils just applying force on the cube and hand and it can cope with all of those perturbations and still solve the problem although a bit harder as they showed okay now for the last but not least I'm gonna show you some metal learning that eventually happened so when they say metal learning they are not thinking of metal learning in the classical sense like solving multiple tasks they're still solving just a Rubik's Cube or they're just reorienting the cube what I mean is that they learn how to estimate this transition probability function and that's also called online system identification in some other communities so basically that means it learns how to estimate the parameters such as friction gravity implicitly and let me show you the results so they run the policy in a simulation on this flipping problem and they show that once they so we have a LSTM inside of the policy if you remember and they show that once they take the hidden state and just pull it down to zero in these particular moments like it's step 10 and step 30 you can see that the time to complete the flip actually increases and then once the the policy has enough time to again encode the the parameters of the environment it again becomes really fast and then once the next perturbation happens again it gets a bit slower but then it adjusts adapts again so until the hidden state is updated it will be a bit slower but then it just gets to the baseline level the second thing they show is just by resampling the environment dynamics so they let the policy work with certain then a certain environment like specific cube size gravity etc and then they just resample everything like new gravity new cube size new cube mass and they show that again the policy kind of slows down and then again learns the dynamics encodes the dynamics inside of the LSTM hidden cell and yeah it just becomes much better the third thing they did is breaking a random joint and basically what I do is at one point of time they just disable one of the joints so whatever action we send to the joint it won't move because it's dead and they show that after that happens basically it will still learn how to how to solve the problem will be a bit slower and the green line here is just a policy that started with a broken with the same joint broken from the start and you can see that this one is a bit worse and they speculate here I think just a second that we hypothesize that this could be because the policy has already locked in some information in its recurrent state and therefore is not adjustable as adjustable anymore alternatively maybe it just has not accumulated enough information yet and the baseline policy is in the lead because it has an information advantage of at least 10 achieved flips I doubt that's the case because I can see the offset continue like being constant even after 20 steps here so yeah that's that'd be interesting to investigate further still you can understand the high-level picture here basically it's learning how to encode environment parameters inside the LSTM is hidden state which is pretty awesome a couple more things yeah they say here we found very interesting that our policy can learn to adapt internally to broken joints this is in contrast contrast to prior work it explicitly searched over a set of policies until they found one that works for a broken robot so I'm not sure which of the randomizations they applied in the ADR setting helps them learn this thing whether they I think I'm pretty confident they didn't simulate the broken joints inside the training so it's pretty funny that this actually works the final thing I want to show you is this the train the predictor over those LSTM's hidden states to predict whether a specific parameter in a particular environment is higher than the average or lower than the average to make that a bit less abstract so it's trying to predict whether maybe the gravity or the cube mass is higher than the average over all of the environments was trained on or lower so they just use a cross entropy loss and to it outputs one when it's certain that the parameter is higher than the average otherwise it tries to output zero and you can see here maybe for the cube size the range is this and this is the average and similarly for the Rubik's Cube problem and once it was trained like that so on a bunch of different environments we take the LSTM's hidden state we have a simple predictor I think they just use a simple fully connected layer and finally the sigmoid output so once they train it on a bunch of those environments with the true labels known they deployed and they showed that here for example on the block orientation policy you can see that certain parameters such as cube size the the the the actual LSTM hidden states is informed enough so that the policy that was trained can learn to with 85% accuracy predict that it's higher than the average which means there is enough information inside of the LSTM state to do this and they notice also that for the cube mass the block orientation policy stores more information about it than the Rubik's Cube policies you can see here this the cube mass the purple one has around 60 something accuracy on the block orientation task whereas here it's a bit above 50% which is close to random chance and they keep they say here we hypothesize that this is because the block orientation policy uses a dynamic approach that tosses the block around to flip it in order contrast the Rubik's Cube policy flips the cube much more deliberately in order to avoid unintentional misalignment of the cube faces so that means that basically the sewing Rubik's Cube is it's less important to know the actual cube mass and that's why the LSTM just doesn't encode that information whereas here because flipping is much less deliberate they don't have to be afraid that one of the faces will misalign like this you can just toss it with much more much higher strength so it encodes that information a bit more than the Rubik's Cube policy that's super fascinating think and they say here this is the evidence that the policy successfully inferring and storing useful information regarding the environment parameters in LSTM hidden and cell states know that we do not train the policy explicitly to store information about the semantically meaningful physical parameters and that's really super awesome that's it hopefully you liked the video if you did you know the drill just like subscribe share and hope to see you next time
[{"start": 0.0, "end": 5.5, "text": " What's up? In this video I'm gonna cover this paper called solving Rubik's Cube"}, {"start": 5.5, "end": 11.28, "text": " with a robot hand by the pretty much complete OpenAI team as you can see"}, {"start": 11.28, "end": 17.14, "text": " here and the goal here is to solve the Rubik's Cube using a robotic hand"}, {"start": 17.14, "end": 23.52, "text": " obviously and but the thing I want you to understand is that the point is not"}, {"start": 23.52, "end": 27.12, "text": " to actually figure out how to solve the cube they just use some off-the-shelf"}, {"start": 27.12, "end": 33.64, "text": " solver for that the whole the difficulty in the project lies in sensing the"}, {"start": 33.64, "end": 39.08, "text": " state of the hand the state of the cube and advising a control policy I"}, {"start": 39.08, "end": 43.64, "text": " controlling those joints on the hand in order to solve the cube so it's a"}, {"start": 43.64, "end": 47.72, "text": " control and sensing problem not the actual how to solve the cube problem and"}, {"start": 47.72, "end": 51.760000000000005, "text": " having said that take a look at this video"}, {"start": 51.76, "end": 58.559999999999995, "text": " we're trying to build robots that learn a little bit like humans do by trial"}, {"start": 58.559999999999995, "end": 65.48, "text": " and error what we've done is train an algorithm to solve the Rubik's Cube one"}, {"start": 65.48, "end": 71.32, "text": " handed with a robotic hand which is actually pretty hard even for a human"}, {"start": 71.32, "end": 77.68, "text": " to do we don't tell it how the hand needs to move that the cube in order to"}, {"start": 77.68, "end": 83.04, "text": " get there the particular friction that's on the fingers how easy it is to turn"}, {"start": 83.04, "end": 87.44000000000001, "text": " the faces on the cube what the gravity what the weight of the cube is all of"}, {"start": 87.44000000000001, "end": 91.76, "text": " these things it needs to learn by itself"}, {"start": 92.56, "end": 96.80000000000001, "text": " awesome hopefully you got some high-level understanding from that video"}, {"start": 96.80000000000001, "end": 100.48, "text": " and now let's just dig into the paper it's a quite a long paper hopefully"}, {"start": 100.48, "end": 105.68, "text": " you'll at the end of this video I'll understand it much better okay let's"}, {"start": 105.68, "end": 111.60000000000001, "text": " start with this the two main components that this new paper advised because as a"}, {"start": 111.60000000000001, "end": 116.16000000000001, "text": " side note they previously published this paper called learning gestures in hand"}, {"start": 116.16000000000001, "end": 120.88000000000001, "text": " manipulation where the setup was pretty similar the only difference is that they"}, {"start": 120.88000000000001, "end": 127.72, "text": " are trying to just reorient this block that has these open AI letters on every"}, {"start": 127.72, "end": 132.16, "text": " single face of this cube and so the goal was much simpler you just need to flip"}, {"start": 132.16, "end": 137.32, "text": " the cube and to get from a certain source position into a certain target"}, {"start": 137.32, "end": 143.16, "text": " configuration and that's it it's still a complicated task but solving the Rubik's"}, {"start": 143.16, "end": 149.6, "text": " Cube is just a level up so yeah and the two main components they say are key to"}, {"start": 149.6, "end": 154.72, "text": " solve this problem the Rubik's Cube is this automatic domain randomization"}, {"start": 154.72, "end": 161.04, "text": " algorithm and a new robot platform built from machine learning and high level"}, {"start": 161.04, "end": 166.2, "text": " this automatic domain randomization so they in the previous paper and this one"}, {"start": 166.2, "end": 170.84, "text": " they already used domain randomization but the novelty is this in this"}, {"start": 170.84, "end": 175.12, "text": " automatic part so they are automatically learning how to create progressively"}, {"start": 175.12, "end": 180.79999999999998, "text": " harder and harder environments and by doing that it turns out that the policy"}, {"start": 180.79999999999998, "end": 186.72, "text": " once deployed on a real robotic hand treats the like the real world as just"}, {"start": 186.72, "end": 193.32, "text": " another simulation matrix confirmed and let's dig into the high level"}, {"start": 193.32, "end": 200.52, "text": " explanation so as you can see here the the setup looks something like this so"}, {"start": 200.52, "end": 204.6, "text": " they have bunch of these simulations and you can see every single hand"}, {"start": 204.6, "end": 208.32, "text": " corresponds to different simulation and everything is randomized from the"}, {"start": 208.32, "end": 212.48, "text": " appearance you can see different the color hands the physics like the"}, {"start": 212.48, "end": 217.44, "text": " gravity the cube size the cube mass the friction of the robotic hand bunch of"}, {"start": 217.44, "end": 222.0, "text": " stuff is randomized so once you have the data once you have those simulations"}, {"start": 222.0, "end": 227.04, "text": " what they do is they train two separate pipelines so one is for training the"}, {"start": 227.04, "end": 232.07999999999998, "text": " control policy and you can see it here so it gets as an input it gets the"}, {"start": 232.07999999999998, "end": 237.2, "text": " fingertip position so you get pretty much five 3d points and it also gets the"}, {"start": 237.2, "end": 241.39999999999998, "text": " state of the cube so that's the position the orientation and the six face angles"}, {"start": 241.4, "end": 248.64000000000001, "text": " as we'll soon see and once you have that state of both the hand and the cube"}, {"start": 248.64000000000001, "end": 254.44, "text": " they train this LSTM based policy network in order to output these actions"}, {"start": 254.44, "end": 263.16, "text": " and that will be just 20 times 11 basically 20 means 20 joints so the hand"}, {"start": 263.16, "end": 268.02, "text": " has 20 joints they can control and 11 is just discretized angles because they"}, {"start": 268.02, "end": 272.64, "text": " later show that that's much better and the place is much better once they do"}, {"start": 272.64, "end": 276.84, "text": " that instead of using continuous actions so those are just the relative angles so"}, {"start": 276.84, "end": 282.91999999999996, "text": " maybe move this specific joint by two degrees and that's it on the other hand"}, {"start": 282.91999999999996, "end": 290.4, "text": " the second pipeline takes these three RGB images as the input and it tries to"}, {"start": 290.4, "end": 295.15999999999997, "text": " regress the the cube pose the cube orientation as well as the face angles"}, {"start": 295.16, "end": 299.8, "text": " and we'll see all of the details a bit later once they combine all of that"}, {"start": 299.8, "end": 305.76000000000005, "text": " together we have this one this this chart and you can see here that the cube"}, {"start": 305.76000000000005, "end": 310.72, "text": " pose always comes from the visual system but because it was really tough actually"}, {"start": 310.72, "end": 315.58000000000004, "text": " to figure out the absolute face angles so you have the cube and you have to"}, {"start": 315.58000000000004, "end": 321.04, "text": " figure out which exact angle and you have from minus pi to pi the angle goes"}, {"start": 321.04, "end": 325.72, "text": " from minus pi to pi so it's really hard to regress it because all for the"}, {"start": 325.72, "end": 330.12, "text": " occlusions etc and they sometimes actually use the cube itself it's a"}, {"start": 330.12, "end": 333.8, "text": " smart cube that has sensors and we'll see that in a moment and it basically"}, {"start": 333.8, "end": 339.6, "text": " just outputs and gives to the policy these numbers which represent the the"}, {"start": 339.6, "end": 345.0, "text": " face angles so they either use vision or they use the cube itself in order to get"}, {"start": 345.0, "end": 349.16, "text": " the face angles so combine them with the fingertip positions that's the input"}, {"start": 349.16, "end": 353.32000000000005, "text": " observation to the policy and they basically train the LSTM based policy to"}, {"start": 353.32000000000005, "end": 356.68, "text": " output the correct actions in order to solve the cube so that's the high level"}, {"start": 356.68, "end": 361.46000000000004, "text": " explanation now let's start digging into details they say here the the learning"}, {"start": 361.46000000000004, "end": 364.96000000000004, "text": " requires vast amount of training data which is hard and expensive to acquire"}, {"start": 364.96000000000004, "end": 368.0, "text": " on a physical system I should have mentioned this a bit earlier but"}, {"start": 368.0, "end": 372.20000000000005, "text": " basically you have two options you can either train this thing on a robot but"}, {"start": 372.20000000000005, "end": 376.72, "text": " because we're trying to learn we know we'll need a lot of data and it's simply"}, {"start": 376.72, "end": 381.08000000000004, "text": " not practical to train the policy on a physical robot and in the previous"}, {"start": 381.08000000000004, "end": 387.64000000000004, "text": " paper the system was called Dactyl just FYI basically they showed that the hand"}, {"start": 387.64000000000004, "end": 392.64000000000004, "text": " had a lot of breakages so it's super impractical to train this even if you"}, {"start": 392.64000000000004, "end": 396.6, "text": " have a farm of robots and this thing this this hand is pretty expensive I can"}, {"start": 396.6, "end": 400.8, "text": " assume so it's really hard to have a setup to collect enough data to train"}, {"start": 400.8, "end": 403.84000000000003, "text": " the policy so they have to go to simulation but once you're in a"}, {"start": 403.84, "end": 409.0, "text": " simulation you need to advise a good way to bridge the domain gap the reality"}, {"start": 409.0, "end": 414.67999999999995, "text": " gap to in order to have the policy training in sim be good enough in the"}, {"start": 414.67999999999995, "end": 420.88, "text": " actual real world so that's just something to keep in mind first of all"}, {"start": 420.88, "end": 425.12, "text": " let me just briefly explain so the goals the first goal we saw was the block"}, {"start": 425.12, "end": 429.35999999999996, "text": " reorientation so that was in there in the previous paper they were solving"}, {"start": 429.35999999999996, "end": 433.53999999999996, "text": " this one and they're using it as a baseline in this one so the goal is"}, {"start": 433.54, "end": 439.56, "text": " considered solved if the orientation of the cube falls inside this 0.4 radians"}, {"start": 439.56, "end": 445.68, "text": " from the target orientation and that's something like 22 degrees more or less"}, {"start": 445.68, "end": 451.88, "text": " 22 23 degrees so it's a pretty big margin but yeah they just figure out"}, {"start": 451.88, "end": 456.44, "text": " that number works for them okay now you can see here visualized so this is the"}, {"start": 456.44, "end": 461.14000000000004, "text": " the block reorientation task you can see the target orientation here and this is"}, {"start": 461.14, "end": 465.4, "text": " the Rubik's Cube task and you can see the target is the solved cube in this"}, {"start": 465.4, "end": 473.12, "text": " case they found that one interesting thing to notice here is that it's really"}, {"start": 473.12, "end": 478.12, "text": " hard to rotate certain once you have a single hand it's really hard to rotate"}, {"start": 478.12, "end": 484.88, "text": " any face other than the top face and they say here we found rotating the top"}, {"start": 484.88, "end": 488.62, "text": " face to be far simpler than rotating other faces so they said it's really"}, {"start": 488.62, "end": 493.04, "text": " hard to rotate this one if you have only a single hand so it's much easier to"}, {"start": 493.04, "end": 498.32, "text": " rotate the top face and thus instead of rotating arbitrary faces we combine"}, {"start": 498.32, "end": 503.12, "text": " together a flip and the top face rotation in order to perform the desired"}, {"start": 503.12, "end": 507.92, "text": " operation that these sub goals can then be performed sequentially to eventually"}, {"start": 507.92, "end": 511.96, "text": " solve the Rubik's Cube so just using those two operations the flip and the"}, {"start": 511.96, "end": 515.6800000000001, "text": " face rotation the top face rotation they can solve the cube when it comes to"}, {"start": 515.68, "end": 518.68, "text": " solving the Rubik's Cube competing a solution sequence can easily be done"}, {"start": 518.68, "end": 522.68, "text": " with existing software libraries like a Siam Bo solver so that's what I mentioned"}, {"start": 522.68, "end": 527.2399999999999, "text": " in the beginning of the video they use off-the-shelf solver and in this work"}, {"start": 527.2399999999999, "end": 531.1999999999999, "text": " the key problem is thus in sensing and control not finding the solution"}, {"start": 531.1999999999999, "end": 537.1999999999999, "text": " sequence so keep that in mind okay briefly just let me just go through this"}, {"start": 537.1999999999999, "end": 542.0799999999999, "text": " physical setup so they have the cage they have these three cameras one two"}, {"start": 542.08, "end": 547.2800000000001, "text": " three three RGB cameras they have this system called phase space which they use"}, {"start": 547.2800000000001, "end": 551.72, "text": " in order to track the fingertip positions and you can see these small"}, {"start": 551.72, "end": 557.8000000000001, "text": " LEDs on the fingertips and that's what a what this phase space system motion"}, {"start": 557.8000000000001, "end": 561.72, "text": " motion capture system uses in order to figure out the exact 3d positions of"}, {"start": 561.72, "end": 567.6400000000001, "text": " those fingertips so then we have the hand and we have the like the cage that"}, {"start": 567.64, "end": 573.4, "text": " holds everything together so one thing I actually when I first saw this video a"}, {"start": 573.4, "end": 576.64, "text": " couple of years ago I thought that the hand was also developed by open AI I"}, {"start": 576.64, "end": 581.48, "text": " wasn't sure which part of the project they actually developed but actually the"}, {"start": 581.48, "end": 585.12, "text": " the hand itself is from some other company it's called the shadow robot"}, {"start": 585.12, "end": 590.88, "text": " company and it's a pretty marvelous piece of engineering itself the hand itself I"}, {"start": 590.88, "end": 595.06, "text": " think has 24 degrees of freedom so it's a really like high dimensional control"}, {"start": 595.06, "end": 601.1999999999999, "text": " problem so previously nobody could solve nobody could imagine solving like rubik's"}, {"start": 601.1999999999999, "end": 607.64, "text": " cube using like a hand as complex as this one but some other systems exist"}, {"start": 607.64, "end": 612.0, "text": " but they are not learned and they are not like a humanoid hand they are just"}, {"start": 612.0, "end": 614.8, "text": " like some kind of a machine that can solve the cube really fast but that's"}, {"start": 614.8, "end": 619.2399999999999, "text": " not what we are trying to accomplish here aside from the hand and the setup"}, {"start": 619.2399999999999, "end": 623.88, "text": " there is the cube component and I already mentioned it's there is smart"}, {"start": 623.88, "end": 627.68, "text": " cube involved and they say here we therefore replace some of the components"}, {"start": 627.68, "end": 631.24, "text": " of the original Geekr cube with custom ones in order to achieve a tracking"}, {"start": 631.24, "end": 634.96, "text": " accuracy of approximately five degrees so the original cube only can so we"}, {"start": 634.96, "end": 641.04, "text": " need as I said we need to to calculate these angles and we need some better"}, {"start": 641.04, "end": 645.56, "text": " resolution than just nine degrees so we need to have resolution that's and they"}, {"start": 645.56, "end": 649.64, "text": " they figure out the five degrees works so they had to create a custom cube and"}, {"start": 649.64, "end": 652.92, "text": " that's funny and you can see here and I just highlight the charging here because"}, {"start": 652.92, "end": 658.16, "text": " it's not a simple it's not like dummy let's go that way a cube it's a it's"}, {"start": 658.16, "end": 663.7199999999999, "text": " got like a microcontroller it's got a couple of components in order to figure"}, {"start": 663.7199999999999, "end": 669.0, "text": " out those angles and we'll see that in a moment okay a quick detour into how this"}, {"start": 669.0, "end": 675.36, "text": " exactly works is you have this thing in every single face of the cube and it's"}, {"start": 675.36, "end": 680.04, "text": " basically a variable resistor so here is just circuit on the left side and you"}, {"start": 680.04, "end": 683.4399999999999, "text": " can see this variable resistor here actually corresponds to this thing here"}, {"start": 683.4399999999999, "end": 687.9599999999999, "text": " and this is just a simple circuit you've got resistors we've got the positive"}, {"start": 687.9599999999999, "end": 692.3399999999999, "text": " power rail we've got the ground ie the zero potential in the circuit you have"}, {"start": 692.3399999999999, "end": 697.76, "text": " something called operational amplifier here and what this circuit does you can"}, {"start": 697.76, "end": 701.8399999999999, "text": " treat it as a black box and most people do basically just hold the same"}, {"start": 701.8399999999999, "end": 707.6999999999999, "text": " potentials on the plus and on the negative inputs and all in all you've got"}, {"start": 707.7, "end": 713.5200000000001, "text": " a current going through here and the signal here is basically our times I"}, {"start": 713.5200000000001, "end": 718.72, "text": " where I is the current and our is changing so the analog signal will"}, {"start": 718.72, "end": 722.88, "text": " actually be increasing as the resistance increases and the resistance increases"}, {"start": 722.88, "end": 729.24, "text": " once you're turning the the face particular face and so what I do is they"}, {"start": 729.24, "end": 733.6400000000001, "text": " have a microcontroller here inside of the cube so some microcontroller it's"}, {"start": 733.64, "end": 738.96, "text": " got something called analog to digital converter and basically this signal here"}, {"start": 738.96, "end": 744.68, "text": " is being fed into a DC and it can discretize the signal into some value so"}, {"start": 744.68, "end": 752.36, "text": " maybe let's say this range goes from from 0 to 5 volts when you change the"}, {"start": 752.36, "end": 757.4, "text": " when you rotate the cube from 0 to 360 degrees and you can just discretize that"}, {"start": 757.4, "end": 762.12, "text": " into maybe some of the numbers will be you'll just have some kind of a curve"}, {"start": 762.12, "end": 767.4, "text": " and this is the analog analog voltage and this will be the output from the"}, {"start": 767.4, "end": 772.84, "text": " converter and you can then map those numbers into angles and send them via"}, {"start": 772.84, "end": 777.44, "text": " Bluetooth to your policy and that's the part of the observation that the policy"}, {"start": 777.44, "end": 781.96, "text": " uses in order to control the joints and eventually solve the sub goals and"}, {"start": 781.96, "end": 787.8, "text": " eventually solve the cube so that was a quick detour into electronics it's"}, {"start": 787.8, "end": 790.96, "text": " funny to see a piece of electronics inside of enamel paper actually studied"}, {"start": 790.96, "end": 795.24, "text": " electronics it's pretty satisfying feeling to to see that you can actually"}, {"start": 795.24, "end": 798.88, "text": " leverage some of the knowledge to learn on the faculty yeah this will probably"}, {"start": 798.88, "end": 805.1600000000001, "text": " be the first and the last time I'll ever see it anyways let's continue and let me"}, {"start": 805.1600000000001, "end": 808.76, "text": " tell you one more thing before we start jumping into how the policy is trained"}, {"start": 808.76, "end": 814.48, "text": " and etc so you have the setup but you want to have the simulation to be as as"}, {"start": 814.48, "end": 820.0, "text": " closely to you want your simulation to resemble the real world as closely as"}, {"start": 820.0, "end": 826.76, "text": " possible so they showed here that they added additional modeling to the hand"}, {"start": 826.76, "end": 833.32, "text": " like these pulleys and tendons and doing that they improved the simulated hand"}, {"start": 833.32, "end": 838.96, "text": " and you can see on these charts basically they track one particular joint"}, {"start": 838.96, "end": 844.48, "text": " like maybe this one and they track the its angle and you can see the this curve"}, {"start": 844.48, "end": 851.72, "text": " the green one is you get these curves by taking a control sequence and sending"}, {"start": 851.72, "end": 855.0, "text": " that control sequence both to the robotic hand to the real robotic hand"}, {"start": 855.0, "end": 858.96, "text": " and to the simulation and you're just tracking the 3d the angle of that"}, {"start": 858.96, "end": 862.72, "text": " particular joint and you can see there is a discrepancy before the calibration"}, {"start": 862.72, "end": 866.88, "text": " before this modeling was done once they included that you can see that the curves"}, {"start": 866.88, "end": 873.04, "text": " align which means we have much better simulation and thus we'll have better"}, {"start": 873.04, "end": 876.56, "text": " control policy once we deploy it on a real robot and they show this is really"}, {"start": 876.56, "end": 883.48, "text": " important part of their of their paper okay so let me jump to the cube so it"}, {"start": 883.48, "end": 888.48, "text": " turns out modeling the cube is super hard there is so many details like the"}, {"start": 888.48, "end": 894.7199999999999, "text": " friction the size the mass the lots of forces inside the cube itself so aside"}, {"start": 894.7199999999999, "end": 899.56, "text": " from the forces that act inside of the cube there's this thing called a"}, {"start": 899.56, "end": 903.88, "text": " forgiveness factor so once you have the top face and I rotated maybe by a couple"}, {"start": 903.88, "end": 908.3199999999999, "text": " of degrees I can still take this perpendicular face and rotated and it"}, {"start": 908.3199999999999, "end": 913.64, "text": " will snap and it will work but like if you if you go too far now you can't you"}, {"start": 913.64, "end": 918.5999999999999, "text": " basically can't flip the the perpendicular face so that's the"}, {"start": 918.5999999999999, "end": 922.4799999999999, "text": " forgiveness factor so you have to count all of those tiny details inside your"}, {"start": 922.4799999999999, "end": 926.7199999999999, "text": " model in order to get a realistic representation so this here they"}, {"start": 926.72, "end": 930.9200000000001, "text": " performed a very rudimentary dynamics calibration of the parameters so like"}, {"start": 930.9200000000001, "end": 935.52, "text": " the like the size the friction etc which mojoco which is the engine they're using"}, {"start": 935.52, "end": 940.12, "text": " for rendering allows us to specify in order to roughly match a physical Rubik's"}, {"start": 940.12, "end": 944.48, "text": " Cube our goal was not to get an exact match but rather to have a plausible"}, {"start": 944.48, "end": 950.24, "text": " model as a starting point for domain randomization so again ADR will see that"}, {"start": 950.24, "end": 956.48, "text": " that plays a really important part in this work okay and here we are so"}, {"start": 956.48, "end": 961.84, "text": " and they say here so in this previous paper so that's the deck tile the the"}, {"start": 961.84, "end": 965.16, "text": " previous paper they published we were able to train a control policy and a"}, {"start": 965.16, "end": 969.5600000000001, "text": " vision model in simulation and then transfer both to a real robot through the"}, {"start": 969.5600000000001, "end": 974.32, "text": " use of domain randomization however this required a significant amount of manual"}, {"start": 974.32, "end": 978.5600000000001, "text": " tuning and a tight iteration loop between randomization design in"}, {"start": 978.5600000000001, "end": 983.2, "text": " simulation and validation on a robot in this section we describe how automatic"}, {"start": 983.2, "end": 987.5600000000001, "text": " domain randomization can be used to automate this process so it's a"}, {"start": 987.5600000000001, "end": 992.0400000000001, "text": " mouthful basically what it means is a following so in previous paper they for"}, {"start": 992.0400000000001, "end": 996.2800000000001, "text": " every single parameter that they want to randomize like maybe the cube size they"}, {"start": 996.2800000000001, "end": 1002.5600000000001, "text": " had to manually find the range so they had to find maybe this is a 5.4"}, {"start": 1002.5600000000001, "end": 1011.5600000000001, "text": " centimeters this may be 5.9 and once they said those basically margins you can"}, {"start": 1011.56, "end": 1017.3199999999999, "text": " sample anything in between but you can't sample outside that range so that means"}, {"start": 1017.3199999999999, "end": 1022.0799999999999, "text": " you have to for every single range you have to test whether that actually leads"}, {"start": 1022.0799999999999, "end": 1026.32, "text": " to a better policy when you deploy it into a robot because if you make this"}, {"start": 1026.32, "end": 1031.48, "text": " arbitrary big maybe this goes to 7.2 now the policy will actually be much worse"}, {"start": 1031.48, "end": 1035.8799999999999, "text": " because it's really hard for the policy to learn how to solve some of those cubes"}, {"start": 1035.8799999999999, "end": 1041.36, "text": " and then it leads to a worse policy which is not desirable so what the ADR"}, {"start": 1041.36, "end": 1046.24, "text": " basically does is it starts from a single value so they measure the"}, {"start": 1046.24, "end": 1050.7199999999998, "text": " realistic cube and that was 5.7 actually in this case and then they slowly"}, {"start": 1050.7199999999998, "end": 1056.04, "text": " expand to the left and to the right and they end up with some a bit bigger range"}, {"start": 1056.04, "end": 1059.8, "text": " and then a bit bigger range and so that's kind of an implicit curriculum"}, {"start": 1059.8, "end": 1063.6799999999998, "text": " learning and we'll see those details so that's the whole point so for every"}, {"start": 1063.6799999999998, "end": 1067.52, "text": " single randomization parameter like maybe the like the friction the cube"}, {"start": 1067.52, "end": 1074.36, "text": " mass the cube size even the gravity etc they gradually and automatically kind of"}, {"start": 1074.36, "end": 1079.84, "text": " increased the actual range for that parameter and that's the whole idea the"}, {"start": 1079.84, "end": 1083.96, "text": " high-level explanation of ADR they say it here the main hypothesis that"}, {"start": 1083.96, "end": 1087.6399999999999, "text": " motivates ADR is the training on a maximally diverse distribution of"}, {"start": 1087.6399999999999, "end": 1092.68, "text": " environments leads to transfer via emergent metal learning more concretely"}, {"start": 1092.68, "end": 1096.8799999999999, "text": " if the model has some form of memory and they use LSTMs and we'll see that in"}, {"start": 1096.88, "end": 1100.8000000000002, "text": " a moment it can learn to adjust its behavior during deployment to improve"}, {"start": 1100.8000000000002, "end": 1105.7600000000002, "text": " performance on the current environment over time I by implementing a learning"}, {"start": 1105.7600000000002, "end": 1110.7600000000002, "text": " algorithm internally so basically what happens is that you have a finite"}, {"start": 1110.7600000000002, "end": 1116.0, "text": " capacity policy and you've got pretty much infinite number of sampled"}, {"start": 1116.0, "end": 1121.5200000000002, "text": " environments so that policy can't can't memorize every single solution to every"}, {"start": 1121.5200000000002, "end": 1125.96, "text": " single environment and there is also not a single robust policy that can solve"}, {"start": 1125.96, "end": 1132.16, "text": " each of these really in a performant way so it has to use the memory it has in"}, {"start": 1132.16, "end": 1137.56, "text": " order to adapt to the given environment and improve its skills that way and they"}, {"start": 1137.56, "end": 1142.96, "text": " later show that actually that happens and the policy learns how to encode the"}, {"start": 1142.96, "end": 1147.52, "text": " environment parameters like cube size cube mass gravity etc inside the hidden"}, {"start": 1147.52, "end": 1152.48, "text": " state of an LSTM and that will be an important thing they'll want to show you"}, {"start": 1152.48, "end": 1157.3600000000001, "text": " okay so here is a high-level explanation and I already gave you here so they are"}, {"start": 1157.3600000000001, "end": 1162.6, "text": " basically just updating those ranges and they say here at its core ADR realizes"}, {"start": 1162.6, "end": 1166.32, "text": " a training curriculum that gradually expands the distribution of environments"}, {"start": 1166.32, "end": 1169.96, "text": " for which the model can perform well the initial distribution of environments is"}, {"start": 1169.96, "end": 1174.44, "text": " concentrated on a single environment for example in policy training the initial"}, {"start": 1174.44, "end": 1178.24, "text": " environment is based on calibration values measured from the physical robot"}, {"start": 1178.24, "end": 1183.2, "text": " so they measure for example the cube so it's 5.7 centimeters the gravity is like"}, {"start": 1183.2, "end": 1187.76, "text": " 9.81 whatever so they have these initial values and then they gradually expand"}, {"start": 1187.76, "end": 1191.44, "text": " them and I'll show you in a blog in a second how that looks like here is a"}, {"start": 1191.44, "end": 1196.08, "text": " really nice visualization on their blog you can see that the x-axis is the"}, {"start": 1196.08, "end": 1202.0, "text": " training the timeline basically and the y-axis is the the range I was mentioning"}, {"start": 1202.0, "end": 1206.64, "text": " so they start with 5.7 centimeters but but as the time progresses the ADR does"}, {"start": 1206.64, "end": 1211.96, "text": " its job and the cube eventually comes to this huge range so they actually sample"}, {"start": 1211.96, "end": 1217.4, "text": " some of the cubes will have 5.47 centimeters size and some will have 6.13"}, {"start": 1217.4, "end": 1221.48, "text": " so that makes it much harder to solve the tasks but because they gradually"}, {"start": 1221.48, "end": 1228.0800000000002, "text": " went to that point the policy learns how to cope with those cube size ranges"}, {"start": 1228.0800000000002, "end": 1232.48, "text": " getting back to the paper let me explain the pseudocode for this algorithm but"}, {"start": 1232.48, "end": 1235.8600000000001, "text": " previous prior to that let me just explain this briefly these two equations"}, {"start": 1235.86, "end": 1241.9199999999998, "text": " so they've got this distribution here and the entropy so as what they did is"}, {"start": 1241.9199999999998, "end": 1245.7199999999998, "text": " for every single parameter that they want to randomize like like the friction"}, {"start": 1245.7199999999998, "end": 1249.56, "text": " and gravity they'll have a uniform distribution so for the cube size for"}, {"start": 1249.56, "end": 1256.02, "text": " example 5.5 I don't know like maybe 5.9 they'll just have those ranges and"}, {"start": 1256.02, "end": 1262.04, "text": " they'll have a uniform distribution here so and that they'll have the same thing"}, {"start": 1262.04, "end": 1267.6399999999999, "text": " for every single parameter so this is maybe like cube size blah blah blah and"}, {"start": 1267.6399999999999, "end": 1272.76, "text": " they'll have like friction etc so the final distribution will just be a"}, {"start": 1272.76, "end": 1278.68, "text": " product 1 to D where D is the number of those parameters and they'll just"}, {"start": 1278.68, "end": 1284.08, "text": " multiply all of these uniform distributions fi low means the the"}, {"start": 1284.08, "end": 1290.8, "text": " lower the lower part of the range so that's 5.5 for the cube and the high"}, {"start": 1290.8, "end": 1296.24, "text": " part means the high threshold and that's it and once you have this"}, {"start": 1296.24, "end": 1301.2, "text": " distribution you can find its entropy and for uniform distributions it's quite"}, {"start": 1301.2, "end": 1306.6399999999999, "text": " easy so the entropy is basically in general's case for a discrete entropy"}, {"start": 1306.6399999999999, "end": 1312.8, "text": " for discrete distribution you'll have a simple sum of probability of that event"}, {"start": 1312.8, "end": 1321.24, "text": " times the logarithm of 1 over the probability of the event and this part"}, {"start": 1321.24, "end": 1325.56, "text": " the log part is actually the self-information so it's sometimes denoted"}, {"start": 1325.56, "end": 1330.9199999999998, "text": " like I of X and this is how you calculate the entropy and basically"}, {"start": 1330.9199999999998, "end": 1335.3999999999999, "text": " because the uniform distribution always has the same values so maybe if we take"}, {"start": 1335.3999999999999, "end": 1342.12, "text": " a simple discrete distribution like this one let's say we have three possible"}, {"start": 1342.12, "end": 1349.0, "text": " events so all of them will have 1 over 3 probability and so you can see so the"}, {"start": 1349.0, "end": 1355.7399999999998, "text": " probability is 1 over 3 and then you have the log 1 over 1 over 3 so that's"}, {"start": 1355.7399999999998, "end": 1362.12, "text": " 3 just 3 and we have that 3 times so these will just cancel out and you end"}, {"start": 1362.12, "end": 1369.76, "text": " up with log 3 so in general case this is just the difference they denote"}, {"start": 1369.76, "end": 1374.32, "text": " here so in the continuous case it will just be your entropy will be the"}, {"start": 1374.32, "end": 1386.92, "text": " integral pi of X log 1 over pi of X DX and because for a particular uniform"}, {"start": 1386.92, "end": 1392.36, "text": " distribution this is constant you can just pull it out here and so we've got"}, {"start": 1392.36, "end": 1398.24, "text": " integral over DX and that will just be the size of your domain so maybe here"}, {"start": 1398.24, "end": 1402.44, "text": " it will be whatever the difference between these two are and that's why"}, {"start": 1402.44, "end": 1407.44, "text": " they have the difference here so this thing is just the entropy of a uniform"}, {"start": 1407.44, "end": 1411.28, "text": " distribution that was a long way to say that thing and then they just divide by"}, {"start": 1411.28, "end": 1414.4, "text": " the number of distributions by the number of randomized parameters and you"}, {"start": 1414.4, "end": 1419.84, "text": " get kind of a average entropy and so the the the more spread out these"}, {"start": 1419.84, "end": 1423.16, "text": " distributions are the higher this entropy will be because you can see as"}, {"start": 1423.16, "end": 1428.64, "text": " the as the range increases the entropy increases and so this is a measure of"}, {"start": 1428.64, "end": 1434.2, "text": " how actually demanding the environment the current environment is so that's"}, {"start": 1434.2, "end": 1440.5600000000002, "text": " just a simple way to do that okay here is the actual algorithm they start with"}, {"start": 1440.5600000000002, "end": 1444.88, "text": " some initial calibration values as I mentioned that may be 5.7 centimeters"}, {"start": 1444.88, "end": 1450.0800000000002, "text": " for the cube 9.81 for the gravity and they keep these buffers for every"}, {"start": 1450.08, "end": 1455.04, "text": " single one randomized parameter and they keep the lower and the higher so one for"}, {"start": 1455.04, "end": 1458.08, "text": " the lower range part of range and one for the higher part of the range and"}, {"start": 1458.08, "end": 1463.72, "text": " they keep these these are just thresholds for how performant the policy"}, {"start": 1463.72, "end": 1469.52, "text": " is given a specific environment and M just means so M is just a number of"}, {"start": 1469.52, "end": 1476.04, "text": " iterations we need in order to decide whether we want to expand or squish the"}, {"start": 1476.04, "end": 1481.48, "text": " range of that particular randomized parameter okay so we sample a particular"}, {"start": 1481.48, "end": 1488.24, "text": " vector from the current distribution and we pick random randomized parameters so"}, {"start": 1488.24, "end": 1493.1599999999999, "text": " one of those D parameters that we are using we pick one of them we then sample"}, {"start": 1493.1599999999999, "end": 1500.8, "text": " from X from the uniform distribution and with 50% chance will either pick the"}, {"start": 1500.8, "end": 1510.48, "text": " lower so we'll set lambda I to the lower extreme or with 50% chance will set"}, {"start": 1510.48, "end": 1514.48, "text": " lambda I to a higher extreme so what that means in that particular case with"}, {"start": 1514.48, "end": 1519.2, "text": " the Rubik's Cube that means that sometimes so basically once you sample"}, {"start": 1519.2, "end": 1525.6399999999999, "text": " this you'll sometimes take your sample 5.6 blah blah but like in this case if"}, {"start": 1525.6399999999999, "end": 1530.04, "text": " you pick the lower one you'll set that lambda I that corresponds to the Rubik"}, {"start": 1530.04, "end": 1538.8, "text": " Cube size to 5.5 or to 5.9 so that's this part and basically then you evaluate"}, {"start": 1538.8, "end": 1544.72, "text": " the performance of the policy on that specific environment so hopefully"}, {"start": 1544.72, "end": 1548.56, "text": " understand what happens here so you pick all of the parameters or all of those"}, {"start": 1548.56, "end": 1553.24, "text": " the randomized parameters you have randomly from their ranges but you pick"}, {"start": 1553.24, "end": 1559.52, "text": " one specific one and you intentionally pull it to some extreme either to the"}, {"start": 1559.52, "end": 1563.2, "text": " lower extreme or to the higher extreme and you evaluate the performance you"}, {"start": 1563.2, "end": 1567.24, "text": " save the performance inside of the buffer and you keep doing that until you see"}, {"start": 1567.24, "end": 1572.24, "text": " this threshold M so that may be 128 times once you have enough samples you're"}, {"start": 1572.24, "end": 1576.92, "text": " you're confident that your estimation will be good and you just take the"}, {"start": 1576.92, "end": 1581.72, "text": " average performance you just this is just clearing the buffer and if the"}, {"start": 1581.72, "end": 1586.24, "text": " average performance is good enough whatever this TH is depending on whether"}, {"start": 1586.24, "end": 1591.4, "text": " you're training vision vision network or the policy network so if you're good"}, {"start": 1591.4, "end": 1595.2, "text": " enough that means that the network that was that is being trained doesn't have"}, {"start": 1595.2, "end": 1598.64, "text": " any difficulty solving this particular environment so we can expand the range"}, {"start": 1598.64, "end": 1603.16, "text": " and next time the environment will be even harder on the other hand on the on"}, {"start": 1603.16, "end": 1608.4, "text": " the flip side basically if the performance is really poor that means"}, {"start": 1608.4, "end": 1613.32, "text": " that pulling this particular parameter to its extreme is really hard and we"}, {"start": 1613.32, "end": 1617.4399999999998, "text": " want to actually squish this this uniform distribution for that particular"}, {"start": 1617.4399999999998, "end": 1621.2, "text": " randomized parameter so hopefully this was clear enough let me know in the"}, {"start": 1621.2, "end": 1624.52, "text": " comments because this is arguably the most important algorithmic innovation"}, {"start": 1624.52, "end": 1628.2, "text": " they had in this paper other than that there was a lot of engineering and"}, {"start": 1628.2, "end": 1634.8799999999999, "text": " nothing less important but yeah okay let's continue here is just a high level"}, {"start": 1634.8799999999999, "end": 1640.48, "text": " overview of how the vision pipeline and how the policy pipelines are trained you"}, {"start": 1640.48, "end": 1646.08, "text": " have the model parameters you have the current PHYs so those are the the the"}, {"start": 1646.08, "end": 1650.68, "text": " ranges we've just mentioned for every single randomized parameter and you you"}, {"start": 1650.68, "end": 1655.48, "text": " basically sometimes sample those you generate the data and you pull push the"}, {"start": 1655.48, "end": 1660.7, "text": " data into this buffer but we sometimes just do the ADR so basically snap one"}, {"start": 1660.7, "end": 1666.16, "text": " of those parameters to its extreme and we sometimes update this PHY to make it"}, {"start": 1666.16, "end": 1671.52, "text": " more challenging or to reduce its difficulty and that's the two things are"}, {"start": 1671.52, "end": 1675.0800000000002, "text": " happening parallel we are constantly improving the the difficulty of"}, {"start": 1675.0800000000002, "end": 1679.6000000000001, "text": " environment and we are constantly generating data and training the model"}, {"start": 1679.6000000000001, "end": 1684.52, "text": " and updating the model parameters so yeah that's how it works and you can see"}, {"start": 1684.52, "end": 1688.92, "text": " one one difference between how they train the vision network compared to"}, {"start": 1688.92, "end": 1695.5600000000002, "text": " the policy is that they here they use both the data that is evaluated as well"}, {"start": 1695.56, "end": 1699.2, "text": " as the actual data pulled from the current parameters and they do that"}, {"start": 1699.2, "end": 1704.24, "text": " because data is really expensive and they need all data I can have for that"}, {"start": 1704.24, "end": 1708.6, "text": " particular policy the problem with with this thing is that sometimes you'll be"}, {"start": 1708.6, "end": 1712.8799999999999, "text": " pulling data that's too hard because this is still remember this is still"}, {"start": 1712.8799999999999, "end": 1717.6399999999999, "text": " under evaluation we still don't know whether snapping that parameter to its"}, {"start": 1717.6399999999999, "end": 1722.8799999999999, "text": " extreme is too hard or not and then yeah there's just the trade-off but they"}, {"start": 1722.88, "end": 1727.0, "text": " figure out it's good enough for their approach okay let's now see what they"}, {"start": 1727.0, "end": 1732.0400000000002, "text": " randomize in their simulations and already mentioned like the physics like"}, {"start": 1732.0400000000002, "end": 1737.3200000000002, "text": " geometry friction gravity etc the interesting part is this one the"}, {"start": 1737.3200000000002, "end": 1741.1200000000001, "text": " adversarial what they do here is the following so they have their training"}, {"start": 1741.1200000000001, "end": 1747.44, "text": " the policy network that's the P and it's outputting those 20 times 11"}, {"start": 1747.44, "end": 1752.0, "text": " categorical distributions that correspond to angles for particular"}, {"start": 1752.0, "end": 1757.28, "text": " joints and on the other hand they have this network the adversary called are"}, {"start": 1757.28, "end": 1765.36, "text": " which just basically outputs random actions and those get added to these"}, {"start": 1765.36, "end": 1771.96, "text": " ones and the policy needs to learn how to complete the sub goal it's given and"}, {"start": 1771.96, "end": 1777.24, "text": " disregard the randomness introduced by the adversary and they tried two things"}, {"start": 1777.24, "end": 1783.24, "text": " actually try to intentionally to develop a true adversary which will try and"}, {"start": 1783.24, "end": 1789.44, "text": " perform the worst actions in order to kind of confuse the policy but they"}, {"start": 1789.44, "end": 1794.08, "text": " realize that just using a random network is actually even better because it"}, {"start": 1794.08, "end": 1800.6, "text": " brings more diversity and later the policy can better generalize once"}, {"start": 1800.6, "end": 1805.52, "text": " deployed on a real robot so why do they do this and the simple answer is the"}, {"start": 1805.52, "end": 1810.16, "text": " following basically once you have the real hand even though the policy says to"}, {"start": 1810.16, "end": 1815.4, "text": " a specific joint move by maybe two degrees the underlying controllers like"}, {"start": 1815.4, "end": 1822.16, "text": " the PID controller will add some noise so it will maybe execute 2.1 degrees"}, {"start": 1822.16, "end": 1826.08, "text": " instead of 2 degrees and then the policy will still have to learn how to deal"}, {"start": 1826.08, "end": 1829.6399999999999, "text": " with that so and that's why they introduced this kind of modeling into"}, {"start": 1829.64, "end": 1835.5600000000002, "text": " the pipeline they also add noise to the observations basically fingertip"}, {"start": 1835.5600000000002, "end": 1839.68, "text": " positions are kind of added like they had a Gaussian noise to to the positions"}, {"start": 1839.68, "end": 1843.4, "text": " etc so everything you can imagine everything that's in this pipeline they"}, {"start": 1843.4, "end": 1847.96, "text": " try and randomize finally the vision like the lighting conditions was the"}, {"start": 1847.96, "end": 1852.92, "text": " lux level the camera positions so remember they have this cage their cage"}, {"start": 1852.92, "end": 1857.96, "text": " and they figure out the positions and use those as the calibration as the"}, {"start": 1857.96, "end": 1863.04, "text": " initial as initial positions and then in the simulation they'll just have some"}, {"start": 1863.04, "end": 1867.4, "text": " volume that will be expanding and they'll be sampling camera positions"}, {"start": 1867.4, "end": 1871.72, "text": " from that volume and that will introduce randomness they'll also experiment with"}, {"start": 1871.72, "end": 1874.8, "text": " the field of view with bunch of different parameters and yeah that's"}, {"start": 1874.8, "end": 1879.24, "text": " pretty much it and I mentioned here so some of these are fixed in a specific"}, {"start": 1879.24, "end": 1883.56, "text": " simulation so they'll just pick the cube size and the mass and they'll freeze"}, {"start": 1883.56, "end": 1887.28, "text": " that but other parameters like the observation noise and adversaries like"}, {"start": 1887.28, "end": 1891.32, "text": " the the thing I just mentioned will be changing with every single step in the"}, {"start": 1891.32, "end": 1897.2, "text": " episode so yeah just keep that in mind and finally training the policy they use"}, {"start": 1897.2, "end": 1901.92, "text": " this thing called PPO proximal policy optimization which is basically"}, {"start": 1901.92, "end": 1906.16, "text": " actor critic method they devised a couple of years ago it's a fairly"}, {"start": 1906.16, "end": 1910.24, "text": " standard RL algorithm I won't dig into too much depth here basically they have"}, {"start": 1910.24, "end": 1913.84, "text": " the value network they also have the policy network the value network tries"}, {"start": 1913.84, "end": 1918.8799999999999, "text": " to predict the expected value and the policy network tries to predict the"}, {"start": 1918.8799999999999, "end": 1923.04, "text": " optimal actions and in that collaboration they eventually learn a"}, {"start": 1923.04, "end": 1928.4399999999998, "text": " really good policy let me just focus briefly on on on these details like the"}, {"start": 1928.4399999999998, "end": 1932.84, "text": " rewards and actions so I already explained how the actions function so"}, {"start": 1932.84, "end": 1936.6799999999998, "text": " basically for every single joint and there are 20 of them they output these"}, {"start": 1936.68, "end": 1943.88, "text": " categorical distributions so maybe something like this and the x-axis is"}, {"start": 1943.88, "end": 1948.8, "text": " like the angle so here this is the highest probability angle so that maybe"}, {"start": 1948.8, "end": 1952.5800000000002, "text": " correspond to five degrees so they'll pick five degrees so those are the"}, {"start": 1952.5800000000002, "end": 1956.24, "text": " actions the rewards they have three types of rewards and they had some"}, {"start": 1956.24, "end": 1960.44, "text": " reward shaping here as you can see so the first component is the non sparse"}, {"start": 1960.44, "end": 1967.88, "text": " component and for example if they are trying to in the case of cube flipping"}, {"start": 1967.88, "end": 1976.48, "text": " if they're trying to get to the target orientation the reward will be the"}, {"start": 1976.48, "end": 1980.6000000000001, "text": " closer you get to the orientation the final orientation the more reward you'll"}, {"start": 1980.6000000000001, "end": 1984.72, "text": " have for each step so if you decrease the distance between those two"}, {"start": 1984.72, "end": 1988.28, "text": " orientations you'll be getting more reward and how they represent the"}, {"start": 1988.28, "end": 1992.44, "text": " orientations is using something called quaternions and those are just it's just"}, {"start": 1992.44, "end": 1996.2, "text": " an extension of the complex number system into four dimensions it sounds"}, {"start": 1996.2, "end": 2000.56, "text": " pretty complex I recently created a post on LinkedIn and I'll link it down in the"}, {"start": 2000.56, "end": 2004.96, "text": " description where I explained the best resources you can get to learn about"}, {"start": 2004.96, "end": 2008.68, "text": " quaternions on the high level you can treat them as a black box it's just a"}, {"start": 2008.68, "end": 2014.3999999999999, "text": " four-tuple basically four numbers and they somehow captured the notion of the"}, {"start": 2014.4, "end": 2018.52, "text": " orientation of the object and they are used all over the place in computer"}, {"start": 2018.52, "end": 2024.72, "text": " graphics and robotics etc so the second reward is a reward of five whenever the"}, {"start": 2024.72, "end": 2029.2800000000002, "text": " goal is achieved so that's a sparse reward as you can as you can assume and"}, {"start": 2029.2800000000002, "end": 2034.96, "text": " finally the penalty of minus 20 when the cube or the block is dropped so that's"}, {"start": 2034.96, "end": 2040.3600000000001, "text": " the reward system they they deploy here is the actual architecture you can see"}, {"start": 2040.3600000000001, "end": 2043.8400000000001, "text": " the as I mentioned the value the value network and the policy network so that's"}, {"start": 2043.84, "end": 2050.24, "text": " the part of the PPO algorithm and you can see the distribution 20 by 11 and"}, {"start": 2050.24, "end": 2056.56, "text": " value has just a single scalar and one thing interesting here is that the the"}, {"start": 2056.56, "end": 2062.16, "text": " the policy network takes the only the noisy observations whereas the value"}, {"start": 2062.16, "end": 2067.44, "text": " network takes both the non noisy observations which are only available in"}, {"start": 2067.44, "end": 2072.88, "text": " the simulation but also it uses the noisy observations and the reason we can"}, {"start": 2072.88, "end": 2077.6, "text": " use that all of the data that's not available on the actual robot is because"}, {"start": 2077.6, "end": 2081.32, "text": " we'll won't be using the value network at all we'll be just using the policy"}, {"start": 2081.32, "end": 2085.84, "text": " network so this is this a symmetric act or critic approach which they use and"}, {"start": 2085.84, "end": 2091.76, "text": " which they devised in some of their previous papers okay here are some of"}, {"start": 2091.76, "end": 2097.52, "text": " the inputs you can see like cube orientation quaternions fingertip"}, {"start": 2097.52, "end": 2102.1600000000003, "text": " positions blah blah blah you can notice that the only the noisy parts go into"}, {"start": 2102.16, "end": 2105.7599999999998, "text": " the policy networks you can see here the check sign so the noisy fingertip"}, {"start": 2105.7599999999998, "end": 2109.96, "text": " positions you can see the check sign here the Q position and the orientation"}, {"start": 2109.96, "end": 2117.3999999999996, "text": " etc okay a couple of interesting things to notice is this and these enable them"}, {"start": 2117.3999999999996, "end": 2122.8799999999997, "text": " to experiment a lot more is this embed and add approach so you can see here"}, {"start": 2122.8799999999997, "end": 2127.3999999999996, "text": " that the observations are always normalized and then embedded into this"}, {"start": 2127.4, "end": 2132.48, "text": " constant 512 dimensional dimensional space and they just summed it up"}, {"start": 2132.48, "end": 2137.28, "text": " summed it up and they apply value and what that enabled them to do is"}, {"start": 2137.28, "end": 2142.08, "text": " experiment with different observations and keep the experience they accumulated"}, {"start": 2142.08, "end": 2149.2400000000002, "text": " over over many many months intact and they say here so more concretely we're"}, {"start": 2149.2400000000002, "end": 2153.1600000000003, "text": " first embed each type of observation separately without any weight sharing"}, {"start": 2153.16, "end": 2158.3199999999997, "text": " into latent space of dimensionality 512 we then combine all inputs by adding"}, {"start": 2158.3199999999997, "end": 2163.16, "text": " the latent representation of each and applying a value non-linearity after the"}, {"start": 2163.16, "end": 2166.6, "text": " main motivation behind this change was to easily add new observation to an"}, {"start": 2166.6, "end": 2171.0, "text": " existing policy and to share embeddings between value and policy network for"}, {"start": 2171.0, "end": 2175.96, "text": " inputs that feed into both so for example if they have fingertips without"}, {"start": 2175.96, "end": 2180.3999999999996, "text": " noise here and fingertips with the noise they'll be sharing these embeddings in"}, {"start": 2180.4, "end": 2186.84, "text": " those cases otherwise they are not sharing the the weights there okay let's"}, {"start": 2186.84, "end": 2192.92, "text": " continue this is super interesting and mind-blowing so they actually train this"}, {"start": 2192.92, "end": 2196.5, "text": " thing they say here we've been training the Rubik's Cube policy continuously for"}, {"start": 2196.5, "end": 2200.92, "text": " several months at this scale while concurrently improving the simulation"}, {"start": 2200.92, "end": 2205.7200000000003, "text": " fidelity the ADR algorithm they've been tuning the hyperparams and they've been"}, {"start": 2205.7200000000003, "end": 2209.2400000000002, "text": " changing the network architecture so the cumulative amount of experience over"}, {"start": 2209.24, "end": 2214.8399999999997, "text": " that period used for training on the Rubik's Cube is roughly 13,000 years and"}, {"start": 2214.8399999999997, "end": 2220.3599999999997, "text": " which is on the same order of magnitude as the 40,000 years they used to train"}, {"start": 2220.3599999999997, "end": 2227.3999999999996, "text": " the open AI 5 the bots they train for Dota 2 game and basically what I want"}, {"start": 2227.3999999999996, "end": 2233.12, "text": " you to have in mind here is that a single simulation step takes is one is 80"}, {"start": 2233.12, "end": 2238.2, "text": " milliseconds so you just divide this number here the number of milliseconds"}, {"start": 2238.2, "end": 2243.2799999999997, "text": " here by this and you'll have the number of steps that happen in the simulation"}, {"start": 2243.2799999999997, "end": 2248.9199999999996, "text": " since they started doing this and it's a huge number okay experimenting is really"}, {"start": 2248.9199999999996, "end": 2251.6, "text": " important obviously they want to keep the experience that they want to ditch"}, {"start": 2251.6, "end": 2258.8399999999997, "text": " the experience they accumulated and so they do one more thing and they say here"}, {"start": 2258.8399999999997, "end": 2263.7599999999998, "text": " so we therefore rarely train experiments from scratch because it's expensive yeah"}, {"start": 2263.7599999999998, "end": 2267.68, "text": " but instead updated existing experiments and initialized from previous"}, {"start": 2267.68, "end": 2273.48, "text": " checkpoints for both the ADR so they both keep those those thresholds for"}, {"start": 2273.48, "end": 2276.96, "text": " each of those ranges for the randomized parameters and they also keep the policy"}, {"start": 2276.96, "end": 2282.7599999999998, "text": " parameters and they did something called behavior cloning in order not to lose"}, {"start": 2282.7599999999998, "end": 2286.56, "text": " experience they implemented something called behavior cloning if you're"}, {"start": 2286.56, "end": 2290.3599999999997, "text": " familiar with imitation learning this is one of the basic approaches there and"}, {"start": 2290.3599999999997, "end": 2295.6, "text": " they just implemented a variant of this dagger algorithm and why they do this is"}, {"start": 2295.6, "end": 2299.62, "text": " that they can iterate with different policy architectures not only the"}, {"start": 2299.62, "end": 2305.88, "text": " observations and still keep the experience intact pretty much intact so"}, {"start": 2305.88, "end": 2309.52, "text": " what how it works is basically you have something called a student network you"}, {"start": 2309.52, "end": 2315.6, "text": " have the teacher network and this one is basically we input the observation out"}, {"start": 2315.6, "end": 2320.7599999999998, "text": " goes the actions and we use these actions to navigate the environment but"}, {"start": 2320.7599999999998, "end": 2325.52, "text": " what we additionally do is because it's got the value function here as well so"}, {"start": 2325.52, "end": 2329.28, "text": " these are the actions and here is a teacher so that's a that's the the best"}, {"start": 2329.28, "end": 2334.28, "text": " network we currently have with a slightly different architecture and this"}, {"start": 2334.28, "end": 2339.0, "text": " is the new network the new architecture and we want to kind of transfer the"}, {"start": 2339.0, "end": 2343.28, "text": " knowledge experience to that new network and so what we do is additionally we"}, {"start": 2343.28, "end": 2347.16, "text": " have the values here and we have the actions here and we just do MSC loss"}, {"start": 2347.16, "end": 2353.24, "text": " between these two so just simple mean square error loss there and we do simple"}, {"start": 2353.24, "end": 2357.8799999999997, "text": " KL divergence between those two and that will make sure that the student"}, {"start": 2357.8799999999997, "end": 2361.7599999999998, "text": " eventually learns to output the same value functions and to output the same"}, {"start": 2361.7599999999998, "end": 2367.56, "text": " actions in particular states as the teacher network and that's what what I"}, {"start": 2367.56, "end": 2374.12, "text": " mean by behavior cloning in this case okay so they say here this is work"}, {"start": 2374.12, "end": 2377.4799999999996, "text": " surprisingly well allowing us to iterate on the policy architecture quickly"}, {"start": 2377.4799999999996, "end": 2381.56, "text": " without losing the accumulated training progress our cloning approach works with"}, {"start": 2381.56, "end": 2385.96, "text": " arbitrary policy architecture changes as long as the action space remains"}, {"start": 2385.96, "end": 2389.7999999999997, "text": " unchanged so the only thing that needs to remain constant is the action space"}, {"start": 2389.7999999999997, "end": 2395.44, "text": " itself so 20 times 11 everything else can change the observations can change"}, {"start": 2395.44, "end": 2400.6, "text": " and the architecture itself can change so I think already mentioned estimating"}, {"start": 2400.6, "end": 2404.92, "text": " the face angles is super hard so what I did is they so they just added these"}, {"start": 2404.92, "end": 2409.7599999999998, "text": " cutouts so that the problem of estimating the angle is a bit easier and"}, {"start": 2409.76, "end": 2414.6800000000003, "text": " they say here so while predicting position and orientation works directly"}, {"start": 2414.6800000000003, "end": 2418.96, "text": " works well we found predicting all six face angles directly to be much more"}, {"start": 2418.96, "end": 2423.0400000000004, "text": " challenging did you have the occlusion even when using a cube with a symmetric"}, {"start": 2423.0400000000004, "end": 2427.4, "text": " center stickers to work around this we decompose face angle prediction into"}, {"start": 2427.4, "end": 2430.92, "text": " several distinct predictions so for the sake of brevity I'll just skip over the"}, {"start": 2430.92, "end": 2435.36, "text": " details but they estimate this thing called active axis and the active face"}, {"start": 2435.36, "end": 2440.48, "text": " angles and the top face angle in order to and then they do some post processing"}, {"start": 2440.48, "end": 2448.1600000000003, "text": " in order to use as a proxy to get the actual face angles and yeah that's what"}, {"start": 2448.1600000000003, "end": 2451.44, "text": " I do with the vision part and you can see the network here so position"}, {"start": 2451.44, "end": 2456.28, "text": " orientation which is a quaternion and these three elements combined with some"}, {"start": 2456.28, "end": 2461.48, "text": " post-processing indirectly gives them the face angle estimations one thing"}, {"start": 2461.48, "end": 2464.56, "text": " you'll notice is there is many components here and you kind of usually"}, {"start": 2464.56, "end": 2468.2, "text": " have to manually specify that when you do the weighted loss you have to"}, {"start": 2468.2, "end": 2472.4, "text": " manually specify those weights so a way around that is using something called"}, {"start": 2472.4, "end": 2477.6, "text": " focal loss and there is this paper which introduced it previously so then the"}, {"start": 2477.6, "end": 2481.66, "text": " main idea is the following so let's say you're doing classification and the true"}, {"start": 2481.66, "end": 2486.04, "text": " label is maybe one and so you're using cross entropy loss your loss will look"}, {"start": 2486.04, "end": 2491.52, "text": " something like this and let's say you output 0.9 and that means the loss is"}, {"start": 2491.52, "end": 2496.84, "text": " pretty small and what they do is the following they just wait using this this"}, {"start": 2496.84, "end": 2502.96, "text": " here this factor here so because the and the minus log P is the cross entropy"}, {"start": 2502.96, "end": 2509.36, "text": " part so this is the the actual weight and so you can see the bigger the P is"}, {"start": 2509.36, "end": 2513.48, "text": " so the closer we are to the correct solution the smaller the weight for this"}, {"start": 2513.48, "end": 2518.4, "text": " loss so that means roughly that if we are performing really good here we want"}, {"start": 2518.4, "end": 2522.44, "text": " ditch we want to drop the weight for that component because we are already"}, {"start": 2522.44, "end": 2527.84, "text": " really good there but if we're maybe struggling here we'll increase the weight"}, {"start": 2527.84, "end": 2532.2400000000002, "text": " that goes with this loss component and that's pretty much it and then they do"}, {"start": 2532.2400000000002, "end": 2535.96, "text": " the ablation studies and they show that using the full model that means using"}, {"start": 2535.96, "end": 2540.52, "text": " the focal loss that the domain randomization etc they achieve the best"}, {"start": 2540.52, "end": 2548.8, "text": " results looking at maybe a real cube you can see that the orientation error is"}, {"start": 2548.8, "end": 2553.36, "text": " seven point something you can see the other values and that's much better than"}, {"start": 2553.36, "end": 2558.96, "text": " when they don't use the randomization or no focal loss etc so yeah you can see"}, {"start": 2558.96, "end": 2565.92, "text": " the order of magnitude of these errors and it's not small but it's good enough"}, {"start": 2565.92, "end": 2570.64, "text": " let's let's continue and jump into results it shows some results on the"}, {"start": 2570.64, "end": 2576.8, "text": " sim to sim transfer performance and basically the model was the policy was"}, {"start": 2576.8, "end": 2581.04, "text": " trained in the ADR setting and the test set also comes from the simulation but"}, {"start": 2581.04, "end": 2586.4, "text": " using some manually specified ranges and so that's the the holdout data and they"}, {"start": 2586.4, "end": 2590.36, "text": " show that as the randomization level increases and that's the entropy I"}, {"start": 2590.36, "end": 2595.64, "text": " mentioned previously as it increases and it's a good measure of how random how"}, {"start": 2595.64, "end": 2600.72, "text": " big those ranges are you can see that the performance on the the number of"}, {"start": 2600.72, "end": 2609.44, "text": " successes on the of that on that holdout set gets better and that's one thing and"}, {"start": 2609.44, "end": 2615.14, "text": " the second thing is they show here the performance on the real data and using"}, {"start": 2615.14, "end": 2622.68, "text": " the ADR so the more randomization they add so you can see the entropy increases"}, {"start": 2622.68, "end": 2627.56, "text": " basically the better the results get if we take this specific policy the XXL"}, {"start": 2627.56, "end": 2631.2799999999997, "text": " policy which was trained for a lot of time you can see they even didn't they"}, {"start": 2631.2799999999997, "end": 2636.3599999999997, "text": " didn't put in the numbers here so you can assume a couple of months and you"}, {"start": 2636.3599999999997, "end": 2642.7999999999997, "text": " can see the the mean and the median number of successes in the block"}, {"start": 2642.7999999999997, "end": 2648.04, "text": " rotation problem increases and it's pretty good in the real world on the"}, {"start": 2648.04, "end": 2653.56, "text": " real robot as well okay let's continue and see what else is there this"}, {"start": 2653.56, "end": 2656.96, "text": " demonstrates that the curriculum learning is really important so the blue"}, {"start": 2656.96, "end": 2662.84, "text": " line is trained the blue policy is trained in the ADR setting whereas all"}, {"start": 2662.84, "end": 2669.32, "text": " of the other curves are manual using the manually set domain randomization ranges"}, {"start": 2669.32, "end": 2673.46, "text": " so what they basically do is the following so when you start with ADR we"}, {"start": 2673.46, "end": 2677.2, "text": " start with a single value and then you slowly expand the range and you expand"}, {"start": 2677.2, "end": 2682.96, "text": " the range again and they expand the range again and this may be like"}, {"start": 2682.96, "end": 2690.72, "text": " the cube size this is the size and how they produce these other curves is for"}, {"start": 2690.72, "end": 2695.8399999999997, "text": " example this one corresponds to maybe this range and the the worst one"}, {"start": 2695.8399999999997, "end": 2699.7599999999998, "text": " actually corresponds to the biggest range so what happens there is that if"}, {"start": 2699.7599999999998, "end": 2705.3599999999997, "text": " you start training the network with a huge range it's actually going to"}, {"start": 2705.36, "end": 2711.44, "text": " underperform because it didn't have the the benefit the the luxury of doing it"}, {"start": 2711.44, "end": 2716.2400000000002, "text": " gradually as it was just thrown into the water so to speak and it was really hard"}, {"start": 2716.2400000000002, "end": 2722.4, "text": " for the for that policy to cope so basically yeah this is a argument for"}, {"start": 2722.4, "end": 2726.36, "text": " using the curriculum learning pretty much here again they showed that on the"}, {"start": 2726.36, "end": 2734.44, "text": " block orientation problem as we increase the ADR ranges the entropy increases and"}, {"start": 2734.44, "end": 2738.2000000000003, "text": " the errors for estimating the orientation and position of the vision"}, {"start": 2738.2000000000003, "end": 2744.64, "text": " pipeline gets better and better you can see this gets much better as we increase"}, {"start": 2744.64, "end": 2750.12, "text": " the entropy similarly for the rub Rubik's Cube the orientation position and"}, {"start": 2750.12, "end": 2757.08, "text": " the top angle all increase going to higher entropy ADRs and that's that"}, {"start": 2757.08, "end": 2763.6, "text": " makes sense okay this is the final result they got on the Rubik's Cube"}, {"start": 2763.6, "end": 2768.52, "text": " basically what I did is they took the maximally hard scramble that means if"}, {"start": 2768.52, "end": 2774.2799999999997, "text": " you know some Rubik's Cube theory at the hardest state you can be in is 26 quarter"}, {"start": 2774.2799999999997, "end": 2781.36, "text": " rotations away from the like completed cube state that means quarter rotation"}, {"start": 2781.36, "end": 2786.08, "text": " is just a 90 degree rotation on the cube and it can be shown that there is only"}, {"start": 2786.08, "end": 2790.7599999999998, "text": " three states that there are 26 quarter rotations away from the goal from the"}, {"start": 2790.76, "end": 2796.0400000000004, "text": " completed cube state and they took one of those hard states and they use those"}, {"start": 2796.0400000000004, "end": 2801.0800000000004, "text": " to test the policy and they can show here they showed here that on the full"}, {"start": 2801.0800000000004, "end": 2806.96, "text": " scramble so and you basically have to have 43 successes to get to the"}, {"start": 2806.96, "end": 2810.8, "text": " completed state because we're only using remember we're just using the phase"}, {"start": 2810.8, "end": 2815.5200000000004, "text": " rotations and we're just using the flipping and basically we need 43"}, {"start": 2815.52, "end": 2821.08, "text": " successes in order to successfully solve that particular scramble and they show"}, {"start": 2821.08, "end": 2828.64, "text": " that for the XXL ADR so highly randomized ADR and using the Giker cube"}, {"start": 2828.64, "end": 2832.32, "text": " for the face angles instead of vision because you remember it's hard to"}, {"start": 2832.32, "end": 2837.0, "text": " estimate those using vision and they show that they have 60% success rate on"}, {"start": 2837.0, "end": 2842.52, "text": " the half scramble and 20% success rate on the full scramble once deployed on a"}, {"start": 2842.52, "end": 2849.2, "text": " real robot and that's pretty amazing still not perfect but like it's pretty"}, {"start": 2849.2, "end": 2855.16, "text": " awesome yeah that's that result and let me continue and show you some more"}, {"start": 2855.16, "end": 2861.04, "text": " awesome results and you can see that using a rubber glove which changes the"}, {"start": 2861.04, "end": 2866.56, "text": " like effectively changes the friction of the environment or using the tide"}, {"start": 2866.56, "end": 2871.64, "text": " fingers or maybe putting the blanket over the cube which will basically"}, {"start": 2871.64, "end": 2879.8399999999997, "text": " render the vision pipeline useless because I can't figure out the position"}, {"start": 2879.8399999999997, "end": 2884.7999999999997, "text": " and orientation or they do some kind of funny perturbations with giraffes and"}, {"start": 2884.7999999999997, "end": 2890.24, "text": " pencils just applying force on the cube and hand and it can cope with all of"}, {"start": 2890.24, "end": 2894.4, "text": " those perturbations and still solve the problem although a bit harder as they"}, {"start": 2894.4, "end": 2901.08, "text": " showed okay now for the last but not least I'm gonna show you some metal"}, {"start": 2901.08, "end": 2904.48, "text": " learning that eventually happened so when they say metal learning they are"}, {"start": 2904.48, "end": 2907.92, "text": " not thinking of metal learning in the classical sense like solving multiple"}, {"start": 2907.92, "end": 2911.72, "text": " tasks they're still solving just a Rubik's Cube or they're just reorienting"}, {"start": 2911.72, "end": 2917.88, "text": " the cube what I mean is that they learn how to estimate this transition"}, {"start": 2917.88, "end": 2923.64, "text": " probability function and that's also called online system identification in"}, {"start": 2923.64, "end": 2927.16, "text": " some other communities so basically that means it learns how to estimate the"}, {"start": 2927.16, "end": 2932.04, "text": " parameters such as friction gravity implicitly and let me show you the"}, {"start": 2932.04, "end": 2938.0, "text": " results so they run the policy in a simulation on this flipping problem and"}, {"start": 2938.0, "end": 2942.8799999999997, "text": " they show that once they so we have a LSTM inside of the policy if you"}, {"start": 2942.8799999999997, "end": 2948.72, "text": " remember and they show that once they take the hidden state and just pull it"}, {"start": 2948.72, "end": 2954.58, "text": " down to zero in these particular moments like it's step 10 and step 30 you can"}, {"start": 2954.58, "end": 2960.6, "text": " see that the time to complete the flip actually increases and then once the the"}, {"start": 2960.6, "end": 2966.36, "text": " policy has enough time to again encode the the parameters of the environment it"}, {"start": 2966.36, "end": 2971.7999999999997, "text": " again becomes really fast and then once the next perturbation happens again it"}, {"start": 2971.7999999999997, "end": 2977.16, "text": " gets a bit slower but then it adjusts adapts again so until the hidden state"}, {"start": 2977.16, "end": 2983.2799999999997, "text": " is updated it will be a bit slower but then it just gets to the baseline level"}, {"start": 2983.28, "end": 2987.96, "text": " the second thing they show is just by resampling the environment dynamics so"}, {"start": 2987.96, "end": 2993.0800000000004, "text": " they let the policy work with certain then a certain environment like specific"}, {"start": 2993.0800000000004, "end": 2997.92, "text": " cube size gravity etc and then they just resample everything like new gravity new"}, {"start": 2997.92, "end": 3002.0800000000004, "text": " cube size new cube mass and they show that again the policy kind of slows down"}, {"start": 3002.0800000000004, "end": 3008.52, "text": " and then again learns the dynamics encodes the dynamics inside of the LSTM"}, {"start": 3008.52, "end": 3013.12, "text": " hidden cell and yeah it just becomes much better the third thing they did is"}, {"start": 3013.12, "end": 3017.7599999999998, "text": " breaking a random joint and basically what I do is at one point of time they"}, {"start": 3017.7599999999998, "end": 3022.32, "text": " just disable one of the joints so whatever action we send to the joint it"}, {"start": 3022.32, "end": 3029.28, "text": " won't move because it's dead and they show that after that happens basically it"}, {"start": 3029.28, "end": 3036.78, "text": " will still learn how to how to solve the problem will be a bit slower and the"}, {"start": 3036.78, "end": 3041.7200000000003, "text": " green line here is just a policy that started with a broken with the same"}, {"start": 3041.7200000000003, "end": 3046.1600000000003, "text": " joint broken from the start and you can see that this one is a bit worse and"}, {"start": 3046.1600000000003, "end": 3053.2400000000002, "text": " they speculate here I think just a second that we hypothesize that this"}, {"start": 3053.2400000000002, "end": 3057.1200000000003, "text": " could be because the policy has already locked in some information in its"}, {"start": 3057.1200000000003, "end": 3061.36, "text": " recurrent state and therefore is not adjustable as adjustable anymore"}, {"start": 3061.36, "end": 3065.6400000000003, "text": " alternatively maybe it just has not accumulated enough information yet and"}, {"start": 3065.64, "end": 3069.52, "text": " the baseline policy is in the lead because it has an information advantage"}, {"start": 3069.52, "end": 3074.8399999999997, "text": " of at least 10 achieved flips I doubt that's the case because I can see the"}, {"start": 3074.8399999999997, "end": 3080.2, "text": " offset continue like being constant even after 20 steps here so yeah that's"}, {"start": 3080.2, "end": 3085.98, "text": " that'd be interesting to investigate further still you can understand the"}, {"start": 3085.98, "end": 3091.74, "text": " high-level picture here basically it's learning how to encode environment"}, {"start": 3091.74, "end": 3095.2, "text": " parameters inside the LSTM is hidden state which is pretty awesome a couple"}, {"start": 3095.2, "end": 3101.12, "text": " more things yeah they say here we found very interesting that our policy can"}, {"start": 3101.12, "end": 3105.72, "text": " learn to adapt internally to broken joints this is in contrast contrast to"}, {"start": 3105.72, "end": 3109.8799999999997, "text": " prior work it explicitly searched over a set of policies until they found one"}, {"start": 3109.8799999999997, "end": 3115.14, "text": " that works for a broken robot so I'm not sure which of the randomizations they"}, {"start": 3115.14, "end": 3121.3599999999997, "text": " applied in the ADR setting helps them learn this thing whether they I think I'm"}, {"start": 3121.36, "end": 3126.04, "text": " pretty confident they didn't simulate the broken joints inside the training so"}, {"start": 3126.04, "end": 3130.7200000000003, "text": " it's pretty funny that this actually works the final thing I want to show"}, {"start": 3130.7200000000003, "end": 3136.2000000000003, "text": " you is this the train the predictor over those LSTM's hidden states to predict"}, {"start": 3136.2000000000003, "end": 3140.7200000000003, "text": " whether a specific parameter in a particular environment is higher than"}, {"start": 3140.7200000000003, "end": 3144.6, "text": " the average or lower than the average to make that a bit less abstract so it's"}, {"start": 3144.6, "end": 3148.44, "text": " trying to predict whether maybe the gravity or the cube mass is higher than"}, {"start": 3148.44, "end": 3153.4, "text": " the average over all of the environments was trained on or lower so they just use"}, {"start": 3153.4, "end": 3159.44, "text": " a cross entropy loss and to it outputs one when it's certain that the"}, {"start": 3159.44, "end": 3164.64, "text": " parameter is higher than the average otherwise it tries to output zero and"}, {"start": 3164.64, "end": 3169.64, "text": " you can see here maybe for the cube size the range is this and this is the"}, {"start": 3169.64, "end": 3175.36, "text": " average and similarly for the Rubik's Cube problem and once it was trained"}, {"start": 3175.36, "end": 3179.04, "text": " like that so on a bunch of different environments we take the LSTM's hidden"}, {"start": 3179.04, "end": 3182.28, "text": " state we have a simple predictor I think they just use a simple fully connected"}, {"start": 3182.28, "end": 3187.84, "text": " layer and finally the sigmoid output so once they train it on a bunch of those"}, {"start": 3187.84, "end": 3194.56, "text": " environments with the true labels known they deployed and they showed that here"}, {"start": 3194.56, "end": 3199.32, "text": " for example on the block orientation policy you can see that certain"}, {"start": 3199.32, "end": 3206.56, "text": " parameters such as cube size the the the the actual LSTM hidden states is"}, {"start": 3206.56, "end": 3213.1200000000003, "text": " informed enough so that the policy that was trained can learn to with 85%"}, {"start": 3213.1200000000003, "end": 3217.0, "text": " accuracy predict that it's higher than the average which means there is enough"}, {"start": 3217.0, "end": 3223.76, "text": " information inside of the LSTM state to do this and they notice also that for"}, {"start": 3223.76, "end": 3227.92, "text": " the cube mass the block orientation policy stores more information about it"}, {"start": 3227.92, "end": 3232.48, "text": " than the Rubik's Cube policies you can see here this the cube mass the purple"}, {"start": 3232.48, "end": 3238.7200000000003, "text": " one has around 60 something accuracy on the block orientation task whereas here"}, {"start": 3238.7200000000003, "end": 3244.7200000000003, "text": " it's a bit above 50% which is close to random chance and they keep they say"}, {"start": 3244.7200000000003, "end": 3248.64, "text": " here we hypothesize that this is because the block orientation policy uses a"}, {"start": 3248.64, "end": 3253.42, "text": " dynamic approach that tosses the block around to flip it in order contrast the"}, {"start": 3253.42, "end": 3258.04, "text": " Rubik's Cube policy flips the cube much more deliberately in order to avoid"}, {"start": 3258.04, "end": 3264.28, "text": " unintentional misalignment of the cube faces so that means that basically the"}, {"start": 3264.28, "end": 3269.2400000000002, "text": " sewing Rubik's Cube is it's less important to know the actual cube mass"}, {"start": 3269.2400000000002, "end": 3273.0, "text": " and that's why the LSTM just doesn't encode that information whereas here"}, {"start": 3273.0, "end": 3278.4, "text": " because flipping is much less deliberate they don't have to be afraid that one of"}, {"start": 3278.4, "end": 3282.52, "text": " the faces will misalign like this you can just toss it with much more much"}, {"start": 3282.52, "end": 3287.4, "text": " higher strength so it encodes that information a bit more than the Rubik's"}, {"start": 3287.4, "end": 3293.28, "text": " Cube policy that's super fascinating think and they say here this is the"}, {"start": 3293.28, "end": 3296.52, "text": " evidence that the policy successfully inferring and storing useful information"}, {"start": 3296.52, "end": 3301.04, "text": " regarding the environment parameters in LSTM hidden and cell states know that we"}, {"start": 3301.04, "end": 3306.04, "text": " do not train the policy explicitly to store information about the semantically"}, {"start": 3306.04, "end": 3310.28, "text": " meaningful physical parameters and that's really super awesome that's it"}, {"start": 3310.28, "end": 3314.0, "text": " hopefully you liked the video if you did you know the drill just like subscribe"}, {"start": 3314.0, "end": 3341.16, "text": " share and hope to see you next time"}]