POET: Paired Open-Ended Trailblazer | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover "Paired Open-Ended Trailblazer (POET): Endlessly
Generating Increasingly Complex and Diverse Learning Environments and Their Solutions" paper.

They simultaneously keep a collection of multiple environments and their associated agents. Environments are occasionally mutated, agents are optimized (using evolutionary search algorithm) w.r.t. their envs. Finally, periodically a competition happens that decides the best agent (out of all agents in the collection) for each env individually. This forms a potent curriculum that leads to agents mastering even tougher envs.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/1901.01753
✅ Kenneth Stanley on open-endedness: https://www.youtube.com/watch?v=dXQPL9GooyI
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro to POET
08:00 Evolutionary search explained
17:15 POET algorithm
18:40 Environment encoding
21:35 Agent and reward modeling
24:35 POET main loop
27:30 Environment mutation algorithm
30:35 Results
33:20 Direct-path curriculum comparison
39:45 POET "unstucks" agents
41:30 Extensions and outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Kevin Stone
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#poet #evolution #openended
 What's cracking guys? In this video I'll be covering an older paper called Paired Open-Ended Trailblazer or POET for short, endlessly generating increasingly complex and diverse learning environments and their solutions by this group of people that used to work at Uber AI labs and you may recognize Kenneth Stanley as the current open-endedness AI lead at OpenAI. So the reason I'm covering this paper is because it's super interesting. I think it's a very nice model of the evolutionary process itself and the main differentiation point between POET and many of the like common and mainstream approaches nowadays is that they do keep like the not a single environment that's that's kind of crafted and it's very hard but they instead keep up set of environments and corresponding agents and they keep on mutating the environments they keep on optimizing the agents for their corresponding environments and additionally they have this transferring where they do like a type of a competition where you try to find the best agent for a particular environment and if it outperforms the current niche champion then you replace the niche champion with that better agent and you keep on repeating this process so that's a high level kind of picture. Now let me show you their video and then we'll get back to the paper. Okay now that you have a rough idea of what's going on in the paper let's take a bit deeper. They say here that while the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions an important question is whether the problems themselves can be generated by algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula so diverse and expanding curricula and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. So let me contrast this this description here with what's usually done in for example reinforcement learning. So usually what you end up with is you have a complex environment like let me denote that as a is this big big square here. It's a complex environment for example Dota or Starcraft or like Go or chess or all of those environments we've used to seeing and we know are very challenging and what I do is they basically train a very specialized agent so that's that's kind of very performant at that particular environment and so it's a very convergent behavior so you have one agent and you're training it and then you converge to a single set of weights and then you can just deploy those that model in the environment and that's it the process stops. So on the other hand what this open-endedness is all about is instead of having this single environment and a single agent that's adapted that environment why not instead keep like a set of a collection of various different environments and let me denote those by different rectangles here like this is one environment and then we have like a second different environment and then we have like a third environment here etc etc and you keep on optimizing corresponding agents for each of these environments as you can see these are very adapt to these particular environments. Okay I'm gonna denote them so yeah they kind of fit perfectly together that's that's that's kind of a pictorial way to for me to to tell you that this agent is very adapt for this particular environment here. So other than having a set of environments and agents which wouldn't be that interesting in and of itself what they do is they occasionally mutate these environments so let me denote that something like this so this environment is gonna be changed it's gonna become maybe a bit different maybe it's gonna have like something like this and then this environment here may become like a bit different after the mutation etc etc and now you keep on optimizing the agents for these novel environments so now you'll end up with an agent that's kind of fits perfectly together into this environment here and the same goes for all of the other environments here so you do this for all of the pairs of environments and agents and the final step in there in this open ended process is to every now and then take all of the agents you have so take all of these agents and basically make them compete on one of the environments so for example you take this one and this one and this one and you see which one of these is the best for let's say for this particular environment and if this one if this like second agent here is better than this one then you're just gonna kill this one and you replace this agent with this one here and that's the those are the high level components that are needed for this open-endedness poet process we'll see more details as I go deeper into the paper but that's that's the rough idea okay let's continue so the results show that poet produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges many of which cannot be solved so that this is an important point here as they'll be comparing poet with these more direct types of solutions so they cannot be solved by direct optimization alone or even through a direct path curriculum building control algorithm we'll see what that means in a second introduced to highlight the critical role of open-endedness in solving ambiguous challenges so as you probably noticed by the very description of this framework I did not specify any particular class optimization algorithms that is going to be used to adapt to to adapt the agents for their corresponding environments what they use in practice here is the evolutionary strategy algorithm so let me start by explaining what those what that class of algorithm is all about for those of you who are not familiar with evolutionary algorithms so the idea is actually fairly simple what do you have is the following so let's imagine we represent our agent as an as a simple MLP so multi-layer perceptron network so let's imagine we have like some input observation and then we have the agent represented as this MLP with multiple hidden layers and then we have the final linear layer and now it comes the output like for example this is the action space of the agent and as you can imagine this contains bunch of learning weights and we can kind of flatten those out into a vector and let's denote that vector as s theta so this is a vector theta and this theta is obviously a multi-dimensional vector depending on how many parameters you've got and we can represent it as a point in space like as a way of visualizing that that theta vector we can represent it as a point in 2d space so obviously because it's multi-dimensional would have to do some type of a dimensionality reduction method in order to plot it like this but you like bear with me this is this is the visualizations I'm going to use so now what evolutionary algorithms do is in their most common form is you fit a Gaussian like on top of this of this theta vector so this is let me just denote it again this is theta vector this is where we start from and then you fit a Gaussian on top of this vector so basically what it means you take it as a mean you take this vector as the mean of the Gaussian and you have some some some basically covariance matrix it's usually just isotropic like Gaussian and so what the idea here is is now you're going to sample randomly sample novel agents so according to the Gaussian distribution so because it's Gaussian is more dense here you're gonna have more agents here you'll have some agents here you'll maybe have one agent here and so on and now what you do is you take all of those like novel random agents and you evaluate them at the environment at hand and you get the associated fitness you get the associated score of that agent in this particular environment at hand so let's now imagine that for example this agent here got a score of 150 this one got a score of let's say 60 this one got a score of like 35 this one like 40 or like I don't know like five or whatnot and you can imagine immediately see that probably this part of environment is for some reason so the weights that are in this part of the space are kind of more valuable for this particular environment and so you intuitively want to move your vector your theta vector in this direction and so how you do that practically in the in these class of evolutionary algorithms is the following so you have associated perturbation vector that you used in order to get to this particular agent from this theta initial state of set of weights so and you have like corresponding perturbation vector for all of these novel agents here in the environment and so what you'll do is a simple in practice you just do a simple weighted average of all of these vectors and you can imagine that after you do that so after you multiply this vector here with with 150 and this vector here with 60 and you do the same for these other vectors you'll end up with a resultant vector that maybe goes that maybe looks like this maybe you end up going in this direction here and so you move and because you have a learning step you'll you won't move too much you'll maybe move like here and so this is your novel theta so this is theta prime okay again re-trating what we've done here is we've we've taken the initial set of weights theta we've fit a Gaussian on top of that theta vector and what I mean by that is basically take the theta as the mean of the Gaussian and you somehow pick the covariance it's usually isotropic Gaussian then you sample n random points from that Gaussian and what you do by doing that is you sample n novel agents and then you evaluate those agents on the environment at hand you get some associated like scores and you just create this weighted sum of all of these perturbation vector and when I when I say a perturbation vector what I mean is basically this vector here so what it took to get from theta so from from from the original like vector here to get to this novel candidate agent so that's the high level understanding of what's going on now let me show you how the formulas look so that you are not confused next time when you see the formula I think it's always good to have this type of grounding and understand like visual understanding of what's going on or like just the semantics behind the notation okay let's first start and understand this this more general formula here where instead of a modeling distribution as a Gaussian you you kind of you parameterize your distribution using the the weights theta so those are the the weights you start from so this vector here and you somehow model that probability distribution and so the idea is so this this tells you the same story I just told you about so this e theta of I just means how well does this theta I agent perform on this particular environment e so this will be a score like for example hundred fifty as we saw before so let me just connect this with the drawing here so this particular set of weights here so this novel candidate vector is going to be theta I and E of theta I so that the performance of that agent on this particular environment is going to be hundred fifty as we saw here okay so that's how those two connect next up let's imagine that under the current probability distribution that's parameterized by theta this agent theta I has the probability of 0.4 and because it has such a high score like intuitively we would love to increase this probability the next time we sample from this probability distribution so optimally when we do a single step of evolutionary strategy algorithm will boost up this one to maybe 0.6 so we'll have more probability put on to on top of this particular set of weights which makes intuitively sense so again this term here tells you what's the what's the fastest way to increase the probability of this particular theta I so that's what gradient does it shows you the direction of the steepest ascent so that we can maximize this this term here and so it's weighted by the the score that that particular agent achieves and then you just do a sum across all of the end samples so and samples are are those random the random samples we take given the the the initial set of weights theta here okay that's how those two connect so that's just a more general notion of this whole evolutionary strategy setup if we assume now that the probability distribution is a Gaussian this is how this expression like transforms you can see again we have a sum across and samples this time we find how performant are these agents so this here is now going to be theta I let me just change the color so this here is a new set of weights called theta I and we just have to so epsilon subscript I is just a sample from the standard Gaussian distribution so this is basically sampling from your Gaussian and you then evaluate the that novel model on the on the environment you get the score and so as you can see here you just use those as the weights for these resultant vectors so those resultant vectors are expect are exactly these dotted lines I showed here and so yeah basically this equation is the formulaic way of describing how evolutionary search breaks okay hopefully that helped you better understand the evolutionary strategy algorithm as a final note this theta plus sigma times epsilon sub I can be denoted as this so it's a it's a Gaussian with mean equal to theta and with covariance matrix equal to sigma identity matrix so you're basically sampling your agent from this distribution here you evaluate and find the score on this environment e and then you use that as the weight to multiply these perturbation vectors epsilon sub eyes okay enough ramble about evolutionary strategy let's continue and explain understand the poet algorithm itself okay let's see how it proceeds so you initialize this list of environment agents as as an empty list you add the initial environment and you add the initial theta so that's just gonna be a random vector and the initial environment is gonna be basically this thing here so they've been using this by by pedal Walker environment except that this one is already fairly complex let me show you how the initial one will probably look like so it will look something like this it's completely flat you don't even have the gap here you you will have like nothing you'll just have a flat environment that's gonna be the initial environment of the poet algorithm okay so let's go back here and then once you add that initial pair of environment environment agent pair so then you do basically capital T number of iterations and in each of those iterations you do the following so every now and then you are going to mutate your environments and so as you can see here so T is the current iteration and every now and then which is dictated by the end mutate you're going to execute this so what does it mean to mutate an environment such as the bipedal by pedal Walker environment we saw here and to understand that let me let me get to this table here so basically each environment is encoded as a as a as a vector and you can treat that vector as a like a gene of the environment and let's understand the format of that of that gene so it looks like something like this so here is our vector that encodes the environment and the first two coordinates here are gonna encode the stump the stump height as you can see here the initial values are gonna be like zero and then 0.4 and that's going to determine the stump heights throughout the environment so just to clarify what this here means is that your stump height so let me draw that as this so here you have like stump height here so stump height and you're gonna have the stump height be between 0 and 0.4 and you're just going to have like a uniform distribution so that means all of these heights here are gonna be of equal probability and throughout the the as the environment progresses you're gonna be sampling from this distribution okay next up you have gap width so the next two coordinates are gonna be gap width and then the next two are gonna be step width and then you're gonna have the step numbers and the roughness let me just kind of quickly explain the semantics behind these so gap width is I guess fairly clear so that's this thing here so this is your gap width this is the stump height then what else they have let me just check so we have step height and number of steps so let me let me see what I can find an environment with steps I think there is one here so you can see here so this is here we have like six steps and you can see what this is the step height so all of these various different parameters that describe the environment are encoded in this in this vector here okay so an important detail here is this mutation step row which tells you how are you going to with each duration which each mutation step how are you going to mutate these these parameters that describe the environment so for this stump height you have 0.2 which means you're going to every now and then with some probability boost up the 0 to like 0.2 and this 0.4 to 0.6 and now you you can imagine you now have a much harder environment because all of the stumps are going to be between like higher on average than than than before and that's how you're progressively and randomly mutating and making environments harder you can do that for all of the parameters that describe the environment here so all across all of these five columns dimensions cool now that we understand the environment let me first before getting back to the algorithm explain how the actual agent is constructed so you can see here how the agent looks like and you basically have four control signals you're controlling the hip torque you're controlling like so here you're controlling the knee torque you're controlling the other hip torque and the other knee torque that's the output so as for the input I think they mentioned it somewhere here let me just if I can zoom out they mentioned here so the agent has 10 lighter range finders for perceiving obstacles and terrain whose measurements are included in the state space in other 14 state variables include hull angle hull angular velocity horizontal and vertical speeds positions of joins blah blah blah blah blah okay so all in all you have its model the following way you have 24 dimensional input observation or state then you have an MLP here a couple of hidden layers and you end up with four dimensional output vector so this is for this is 24 this is just an MLP and that's everything you need to know about the agent the final detail that's important is how do you define the fitness how do you find the score and here is the formulaic description of the reward you can see that if the robot falls then you like the robot gets minus hundred reward so negative reward you can see here that we are encouraging the robot to move like rightward so here Delta X because the X axis basically as you can assume goes from left to right so you want to move as much as possible so that's the Delta as much as possible to the right then it's penalized for the hull angle whereas the hull angle is this thing so basically you want to make sure let me zoom in a bit more so you want to make sure that this particular angle here so this angle here let me know it does maybe alpha you want to minimize it and that means you want to have this this hull being as horizontal as possible and the final detail they have here is they are also penalizing for the applied torque so if we were to explain what this means in human language is it's basically like a like a like a formal description of this informal statement please don't fall move to the right as much as possible try and keep your hull as horizontal as possible and try to minimize the energy consumption while doing all of that so that would be like an informal description of what this this this reward is encoding cool now that we understand all of that let's get back to the algorithm put main loop so we got to this line six here we're somehow mutating the environments we saw the mechanism roughly there is a pseudo algorithm in the appendix I'm going to show you in a minute but like for now just imagine you have like a set of vectors which are encoding for particular environments you can you can kind of imagine you mutate those vectors according to the rules we saw and then you get novel novel environments okay now that we have that environment a novel set of mutated environments we just iterate through those environments and we optimize the agents here as you can see in this step here we're going to take the the environment agent pair we're gonna do a single evolutionary strategy step so this as you remember produces the resultant vector so we just add that resultant vector to the theta that was used as the mean of the Gaussian and we get the novel the novel set of weights for that agent so that's how the agents are optimized here obviously you can see that we have like the learning rate alpha here we have Sigma the standard deviation for the Gaussian at hand and all of that combined goes into the input of this ES step and that's how we update the weights okay finally and I mentioned this every now and then we're going to so every now and then is determined by this end transfer we're going to evaluate candidates as you can see here we take all of the agents except for the theta t plus 1 M except for the Mth agent where M is the current agent in this loop here and we evaluate all of these agents on the Mth environment and we find the best performer for that environment and if the score that the best performer achieved is better than the than the current agents then the niche champion then you replace you basically overwrite so basically bye bye the old agent and here comes the new one which was the best agent which is the most adapt agent for this environment even though it came from some different environment so that's the this interesting cross pollination that's going on okay so on a high level the Poet algorithm is fairly simple you keep this collection of environment agent pairs you keep mutating the environments you keep optimizing the corresponding agents for those particular environments and every now and then you do this cross pollination step whereby for each of the environments you're going to find the best agent overall which may come from some different environment and replace the current niche champion with that one if that happens okay so that's that's everything and now let me just go to appendix and show you the mutation pseudo algorithm in a bit more detail so here we are let's see how it works so we have some list already we iterate through the list of agent environment pairs and we see if they are eligible to reproduce and what this means is they basically evaluate the agent M on this environment M and they see what the score is and if the score is like if it's not in this in this in this range between like for example 50 and 200 they discard that agent for that particular environment I mean both the agent and the environment and why is that well because that means that if you're outside of this range that means that you're either here which means you achieve super low score which means that the environment is probably very hard or it means that you're here which means it's you have a very high score and the environment is very very easy and so you want to kind of filter out those either too hard or too simple of environment after we've done that we have this parent list so basically all of the eligible environment agent pairs are there then we do the environment to produce so this step here is gonna do these random mutations you're gonna take the vector that describes the environment you're gonna do the mutations according to certain rules and you end up with this child list then you again just check this minimal condition IE you check whether they are in this range you discard the ones which are not after all of that is done there is this line 13 where there is something interesting going on so they rank by novelty and how they define novelty is using L2 metric basically they take the encoding vector and they compare it with all of the other environment vectors as well as some of the older like they have this archive of older environments and the further away it is from all of the other environments as defined by L2 the more novel it is so that's how they define the concept of novelty it's it's it's very different compared to the previous environments we've seen and they basically sort this list according to to not according to novelty and once we've done all this they also here they have this evaluation of candidates so they find the best agent for that particular environment and they and if this minimal condition is satisfied then they add the environment agent pair to the list so what can potentially happen here again is that some of the other agents is better at the environment that can this at this child environment and then you're just gonna overwrite the agent with this better agent here so yeah there is a it's kind of intricate but you got the gist of it already you have before I showed you this this pseudo algorithm okay let's get back to the main paper and see where we stopped okay let's get to the results the first comparison they do is they compare poet with the evolutionary strategy from scratch which means you find some fairly complicated environment as the one you can see here and you either train the agent using poet or you directly train evolutionary search so you start from a random agent and you try by random perturbations to find the agent that performs well for this particular environment and it turns out that as soon as the environment gets like decently complicated the evolutionary strategy from scratch starts failing and they mention here some of the failure cases so they say here in effect to obtain positive scores these agents learn to move forward but also to freeze before challenging obstacles which helped them avoid the penalty of minus hundred for falling down and this behavior is a local optimum the agents could in principle learn to overcome the obstacles but instead converge on playing it safe by not moving it's a very interesting behavior like a like a like a risk adversity like being manifested here in these simple agents in the most simple of environments and yeah I guess we see that in the in the real happening in the real world as well so here you can see the results you can see that that that the ES kind of gets stuck here and stays in the same place because otherwise it will fall down and get a minus hundred reward which is suboptimal whereas poet can actually yeah can actually kind of jump across these these gaps and and and continue on doing its job similarly for these other types of environments you can see here as soon as a big obstacle comes here the agent gets stuck whereas the poet keeps on going to the right same here it just stops in front of a big obstacle and does not move and that's the local optimum that they were referring to okay let's continue here you can see that poet can solve even some of very difficult environments and that's that's very very cool results okay here is just the quantitative visualization of how like of this huge gap between poet performance and evolutionary search by the way I'm butchering the names here like I'm not sure whether you can use interchangeably strategy search and like just evolutionary system because I think opening I had some evolutionary strategy paper so it may be the name may be overloaded but I explained the semantics between behind the evolutionary search like a couple minutes ago so hopefully that won't be confusing okay now we've seen the direct comparison between poet and evolutionary search now you can do something a bit more smart and that's doing a curriculum based learning but like not using poet which is highly open-ended but doing a more direct path curriculum learning let's see what what that means so a natural question then is whether the environments created and solved by poet can also be solved by an explicit direct path curriculum building control algorithm the sequence of environment starts with an environment of only flat ground without any obstacles which is easy enough for any randomly initialized agent to quickly learn to complete then each of the subsequent environments are constructed by slightly modifying the current environment okay let me try and explain what that means and what's the difference in approach so on one hand you have poet so you start with some random environment you start with the easy environment and then you randomly start protruding that environment so it's environment number one you do some mutations you get to environment number two and then you do some like mutations you get to three and then you get to four and then etc etc and you end up with environment after n steps which is somewhere here okay so now what they propose is let's start from this very same so so this is the original environment which is the flat ground everything super simple let me just duplicate it here for the sake of easier visualization so we have the same environment so these two here are the same so that was that's the simplest environment possible now instead of having this highly nonlinear mutation path towards M why not just go directly towards N and when I say directly as you know the environment described by a vector and you can literally linearly interpolate between these two vectors and create a set of environments that goes from one to N but following a much more direct path and they mentioned they explained here how exactly they do that they say more specifically to get a new environment each obstacle parameter of the current environment has an equal chance of staying the same value or increasing by corresponding mutation step value in table one until that obstacle parameter reaches that of the target environment so they do introduce a slight stochasticity in this process is not just a pure linear interpolation but like it's very close to that okay so this is the approach they take and let's now see the results they got comparing to this baseline okay here are the results these are something called rose plots and you can see that so run one is a single run of the poet algorithm and as poet was was running you can imagine that the environments were getting harder and harder let me first explain to you how you can parse this this this diagram so each of these five axes here correspond to one of the five parameters we saw in the table before so that means one is coding for like gap one is coding for the stump size one is coding for the roughness etc etc so the hardest environments are the ones which would be touching basically the end points here so these would be the this would be the hardest environment possible so this one here this is the hardest environment okay so looking at these these plots here we can see that so we have here three independent fronts let me first explain what the columns mean and what the what the rows mean so we have three independent runs as and the run like the the environments were getting harder and harder as the as the runs were progressing and then we can notice that the blue pentagons is the direct path curriculum and we can see that it's not able to to achieve the same to solve the same complexity level environments as poet given the similar amount of compute so again connecting this diagram with the thing I draw here basically the the the poet does not achieve never gets to end it maybe gets to like manages to solve like some of the environments the more the more these easier environments but never manages to actually solve these harder environments that's what we can see visually on this on these rose plots here continuing on this is just another representation of that same fact and one thing that's worth mentioning about poet is the following so while poet cannot guarantee reaching a particular preconceived target these results suggested in some environment spaces poets unique ability to generate multiple different challenges and solve them may actually still provide a more promising path towards solving some preconceived target challenges so what this sentence here is telling us is the following so the thing with poet is you can solve arbitrarily complicated environments but like it's not like you can specify like up in front like you cannot upfront specify this particular environment and tell poet hey go and solve this environment which I deeply care about no you can't do that so that's that's that's what this approach here try to do and we show that it failed so the whole point is you have to have this open-ended process and you have to be collecting milestones and maybe some of the milestones turn out to be very useful for the problem you care about so Kenneth Stanley one of the co-authors of this paper built like a whole philosophy behind this this this discovery here he even has a book why greatness cannot be planned where he is discussing this this tension that exists between having clear objectives and where you want to get to versus having this more divergent approach and he's arguing very very very very strongly for this more open-ended approach as opposed to these direct optimization methods and we saw like and he uses results such as this one to to argument why that might be the best thing to do in the long run and that's something that evolution itself is doing it's not like evolution is going towards a particular solution okay let me explain you this very very interesting diagram that that kind of shows the power of the poet algorithm so here you can see the parent environment and this particular agent at iteration 400 learned to do this type of like it's dragging its leg very close to the ground so it's a very we can see it's a very suboptimal behavior so the agent ended up in some local optimum and it cannot get unstuck so what poet does is at one point of time this agent is gonna perform outperform some other agent on some other environment so that's the transfer we saw multiple times and you see that in this environment there are stumps and after training on this so after training this same agent so the same set of weights on this novel environment the agent learned how to straighten up the legs you can see after iteration 1175 the agent is already walking quite like the legs are straightened out and then what happens is after you now copy that agent into the old environment and you keep on iterating you can see that here in the parent environment the agent finally learned how to walk straight so if you did not have this this this dynamics where the agent was sent to another environment trained there and then returned back to this environment you would never get unstuck from this local optimum we see here and so that's that's the whole that's a very powerful thing about poet I guess the main the main con side of poet is you cannot as we saw we cannot directly solve the environments we care about we kind of have to be collecting milestones and hope for what and hope that some of the milestones make sense for what we care about okay let me wrap up this paper by stating the possum some possible extensions of poet although obviously the paper was published in 2019 so I'm not quite sure whether any of these extensions happened or did not okay so they say the following an additional constraint that exists in this initial work is that the body of the agent is fixed ultimately limiting the sort of obstacles it can overcome for example how big of a gap it can jump over here too a more powerful and expressive encoding of morphologies including when I say morphologies they mean body shapes including those like CPP ends they're based on developmental biology could allow us to co-evolve the morphology and the body of the agent along with its brain in addition to the environment it is solving I mean this is a very straightforward extension of the of the research that was presented in this paper you can imagine that not only can you mutate the environment and then adapt the agents to that environment but you can imagine that you can also have a encoding vector for the morphology of the agent and you could be mutating the more that encoding vector and by doing those mutations you can maybe end up with an agent that has like longer legs or you can also imagine that you could be mutating the actual like brain architecture of the agent so that means instead of instead of like having that fixed MLP I described a couple of minutes ago so something like this you can be modifying this architecture as well mutating the architecture so you're mutating the architecture of the brain of the body and you can also imagine you could be mutating the rewards for each of the environments so not why just keep the why keep the reward constant as we saw there like with minus hundred penalty for for falling etc etc let me just find this is a super long paper so you could be also mutating the reward and all of that additional diversity could yeah probably either end up going towards some space that we don't care about or it could be very interesting although you probably would not want to give the reward too much flexibility you want to kind of scope it you roughly know what you care about and then you form a space you maybe form a probability distribution across rewards so that some rewards are more likely than some other ones okay those were some candies for your thought hopefully you liked this this video hopefully found this this idea of poet an open endedness interesting and if you did share the video out consider subscribing join the discord community and until next time bye bye
[{"start": 0.0, "end": 4.48, "text": " What's cracking guys? In this video I'll be covering an older paper called"}, {"start": 4.48, "end": 10.200000000000001, "text": " Paired Open-Ended Trailblazer or POET for short, endlessly generating"}, {"start": 10.200000000000001, "end": 14.16, "text": " increasingly complex and diverse learning environments and their solutions"}, {"start": 14.16, "end": 19.080000000000002, "text": " by this group of people that used to work at Uber AI labs and you may"}, {"start": 19.080000000000002, "end": 24.88, "text": " recognize Kenneth Stanley as the current open-endedness AI lead at OpenAI."}, {"start": 24.88, "end": 28.8, "text": " So the reason I'm covering this paper is because it's super interesting. I think"}, {"start": 28.8, "end": 33.32, "text": " it's a very nice model of the evolutionary process itself and the main"}, {"start": 33.32, "end": 38.36, "text": " differentiation point between POET and many of the like common and mainstream"}, {"start": 38.36, "end": 43.36, "text": " approaches nowadays is that they do keep like the not a single environment"}, {"start": 43.36, "end": 48.36, "text": " that's that's kind of crafted and it's very hard but they instead keep up set"}, {"start": 48.36, "end": 51.96, "text": " of environments and corresponding agents and they keep on mutating the"}, {"start": 51.96, "end": 55.44, "text": " environments they keep on optimizing the agents for their corresponding"}, {"start": 55.44, "end": 59.239999999999995, "text": " environments and additionally they have this transferring where they do like a"}, {"start": 59.239999999999995, "end": 64.44, "text": " type of a competition where you try to find the best agent for a particular"}, {"start": 64.44, "end": 69.2, "text": " environment and if it outperforms the current niche champion then you replace"}, {"start": 69.2, "end": 73.0, "text": " the niche champion with that better agent and you keep on repeating this"}, {"start": 73.0, "end": 77.16, "text": " process so that's a high level kind of picture. Now let me show you their"}, {"start": 77.16, "end": 104.96, "text": " video and then we'll get back to the paper."}, {"start": 197.16, "end": 211.76, "text": " Okay now that you have a rough idea of what's going on in the paper let's take"}, {"start": 211.76, "end": 216.3, "text": " a bit deeper. They say here that while the history of machine learning so far"}, {"start": 216.3, "end": 221.6, "text": " largely encompasses a series of problems posed by researchers and algorithms that"}, {"start": 221.6, "end": 225.12, "text": " learn their solutions an important question is whether the problems"}, {"start": 225.12, "end": 229.44, "text": " themselves can be generated by algorithm at the same time as they are being"}, {"start": 229.44, "end": 234.08, "text": " solved. Such a process would in effect build its own diverse and expanding"}, {"start": 234.08, "end": 238.64000000000001, "text": " curricula so diverse and expanding curricula and the solutions to problems"}, {"start": 238.64000000000001, "end": 243.92000000000002, "text": " at various stages would become stepping stones towards solving even more"}, {"start": 243.92000000000002, "end": 248.0, "text": " challenging problems later in the process. So let me contrast this this"}, {"start": 248.0, "end": 253.4, "text": " description here with what's usually done in for example reinforcement"}, {"start": 253.4, "end": 257.88, "text": " learning. So usually what you end up with is you have a complex environment like"}, {"start": 257.88, "end": 262.6, "text": " let me denote that as a is this big big square here. It's a complex environment"}, {"start": 262.6, "end": 268.22, "text": " for example Dota or Starcraft or like Go or chess or all of those"}, {"start": 268.22, "end": 273.2, "text": " environments we've used to seeing and we know are very challenging and"}, {"start": 273.2, "end": 281.0, "text": " what I do is they basically train a very specialized agent so that's that's kind"}, {"start": 281.0, "end": 285.08, "text": " of very performant at that particular environment and so it's a very"}, {"start": 285.08, "end": 289.12, "text": " convergent behavior so you have one agent and you're training it and then"}, {"start": 289.12, "end": 293.0, "text": " you converge to a single set of weights and then you can just deploy those that"}, {"start": 293.0, "end": 297.76, "text": " model in the environment and that's it the process stops. So on the other hand"}, {"start": 297.76, "end": 301.92, "text": " what this open-endedness is all about is instead of having this single"}, {"start": 301.92, "end": 305.76, "text": " environment and a single agent that's adapted that environment why not"}, {"start": 305.76, "end": 310.72, "text": " instead keep like a set of a collection of various different environments and"}, {"start": 310.72, "end": 314.12, "text": " let me denote those by different rectangles here like this is one"}, {"start": 314.12, "end": 318.6, "text": " environment and then we have like a second different environment and then"}, {"start": 318.6, "end": 325.24, "text": " we have like a third environment here etc etc and you keep on optimizing"}, {"start": 325.24, "end": 329.04, "text": " corresponding agents for each of these environments as you can see these are"}, {"start": 329.04, "end": 334.48, "text": " very adapt to these particular environments. Okay I'm gonna denote them"}, {"start": 334.48, "end": 339.6, "text": " so yeah they kind of fit perfectly together that's that's that's kind of a"}, {"start": 339.6, "end": 346.28000000000003, "text": " pictorial way to for me to to tell you that this agent is very adapt for this"}, {"start": 346.28000000000003, "end": 351.96000000000004, "text": " particular environment here. So other than having a set of environments and"}, {"start": 351.96000000000004, "end": 355.52000000000004, "text": " agents which wouldn't be that interesting in and of itself what they"}, {"start": 355.52000000000004, "end": 359.04, "text": " do is they occasionally mutate these environments so let me denote that"}, {"start": 359.04, "end": 362.8, "text": " something like this so this environment is gonna be changed it's gonna become"}, {"start": 362.8, "end": 367.40000000000003, "text": " maybe a bit different maybe it's gonna have like something like this and then"}, {"start": 367.4, "end": 373.71999999999997, "text": " this environment here may become like a bit different after the mutation etc"}, {"start": 373.71999999999997, "end": 378.08, "text": " etc and now you keep on optimizing the agents for these novel environments so"}, {"start": 378.08, "end": 381.44, "text": " now you'll end up with an agent that's kind of fits perfectly together into"}, {"start": 381.44, "end": 387.35999999999996, "text": " this environment here and the same goes for all of the other environments here"}, {"start": 387.35999999999996, "end": 392.56, "text": " so you do this for all of the pairs of environments and agents and the final"}, {"start": 392.56, "end": 399.36, "text": " step in there in this open ended process is to every now and then take all of the"}, {"start": 399.36, "end": 403.96, "text": " agents you have so take all of these agents and basically make them compete"}, {"start": 403.96, "end": 407.08, "text": " on one of the environments so for example you take this one and this one"}, {"start": 407.08, "end": 410.96, "text": " and this one and you see which one of these is the best for let's say for this"}, {"start": 410.96, "end": 415.96, "text": " particular environment and if this one if this like second agent here is better"}, {"start": 415.96, "end": 421.28, "text": " than this one then you're just gonna kill this one and you replace this agent"}, {"start": 421.28, "end": 426.4, "text": " with this one here and that's the those are the high level components that are"}, {"start": 426.4, "end": 431.46, "text": " needed for this open-endedness poet process we'll see more details as I go"}, {"start": 431.46, "end": 435.23999999999995, "text": " deeper into the paper but that's that's the rough idea okay let's continue so"}, {"start": 435.23999999999995, "end": 439.28, "text": " the results show that poet produces a diverse range of sophisticated"}, {"start": 439.28, "end": 443.53999999999996, "text": " behaviors that solve a wide range of environmental challenges many of which"}, {"start": 443.53999999999996, "end": 447.55999999999995, "text": " cannot be solved so that this is an important point here as they'll be"}, {"start": 447.56, "end": 453.56, "text": " comparing poet with these more direct types of solutions so they cannot be"}, {"start": 453.56, "end": 458.48, "text": " solved by direct optimization alone or even through a direct path curriculum"}, {"start": 458.48, "end": 462.86, "text": " building control algorithm we'll see what that means in a second introduced to"}, {"start": 462.86, "end": 466.72, "text": " highlight the critical role of open-endedness in solving ambiguous"}, {"start": 466.72, "end": 471.86, "text": " challenges so as you probably noticed by the very description of this framework I"}, {"start": 471.86, "end": 475.96, "text": " did not specify any particular class optimization algorithms that is going"}, {"start": 475.96, "end": 481.12, "text": " to be used to adapt to to adapt the agents for their corresponding"}, {"start": 481.12, "end": 484.56, "text": " environments what they use in practice here is the evolutionary strategy"}, {"start": 484.56, "end": 490.4, "text": " algorithm so let me start by explaining what those what that class of algorithm"}, {"start": 490.4, "end": 494.44, "text": " is all about for those of you who are not familiar with evolutionary algorithms"}, {"start": 494.44, "end": 498.4, "text": " so the idea is actually fairly simple what do you have is the following so"}, {"start": 498.4, "end": 504.03999999999996, "text": " let's imagine we represent our agent as an as a simple MLP so multi-layer"}, {"start": 504.04, "end": 508.88, "text": " perceptron network so let's imagine we have like some input observation and"}, {"start": 508.88, "end": 513.52, "text": " then we have the agent represented as this MLP with multiple hidden layers and"}, {"start": 513.52, "end": 517.96, "text": " then we have the final linear layer and now it comes the output like for example"}, {"start": 517.96, "end": 522.2, "text": " this is the action space of the agent and as you can imagine this contains"}, {"start": 522.2, "end": 526.6800000000001, "text": " bunch of learning weights and we can kind of flatten those out into a vector"}, {"start": 526.68, "end": 534.8399999999999, "text": " and let's denote that vector as s theta so this is a vector theta and this theta"}, {"start": 534.8399999999999, "end": 538.1999999999999, "text": " is obviously a multi-dimensional vector depending on how many parameters you've"}, {"start": 538.1999999999999, "end": 543.0799999999999, "text": " got and we can represent it as a point in space like as a way of"}, {"start": 543.0799999999999, "end": 548.4, "text": " visualizing that that theta vector we can represent it as a point in 2d space"}, {"start": 548.4, "end": 551.0799999999999, "text": " so obviously because it's multi-dimensional would have to do some"}, {"start": 551.0799999999999, "end": 555.4399999999999, "text": " type of a dimensionality reduction method in order to plot it like this but"}, {"start": 555.44, "end": 559.8000000000001, "text": " you like bear with me this is this is the visualizations I'm going to use so"}, {"start": 559.8000000000001, "end": 566.84, "text": " now what evolutionary algorithms do is in their most common form is you fit a"}, {"start": 566.84, "end": 571.6400000000001, "text": " Gaussian like on top of this of this theta vector so this is let me just"}, {"start": 571.6400000000001, "end": 576.5600000000001, "text": " denote it again this is theta vector this is where we start from and then you"}, {"start": 576.5600000000001, "end": 583.0, "text": " fit a Gaussian on top of this vector so basically what it means you take it as a"}, {"start": 583.0, "end": 586.56, "text": " mean you take this vector as the mean of the Gaussian and you have some some"}, {"start": 586.56, "end": 591.52, "text": " some basically covariance matrix it's usually just isotropic like Gaussian"}, {"start": 591.52, "end": 597.44, "text": " and so what the idea here is is now you're going to sample randomly sample"}, {"start": 597.44, "end": 601.44, "text": " novel agents so according to the Gaussian distribution so because it's"}, {"start": 601.44, "end": 605.08, "text": " Gaussian is more dense here you're gonna have more agents here you'll have some"}, {"start": 605.08, "end": 610.2, "text": " agents here you'll maybe have one agent here and so on and now what you do is"}, {"start": 610.2, "end": 616.08, "text": " you take all of those like novel random agents and you evaluate them at the"}, {"start": 616.08, "end": 620.2800000000001, "text": " environment at hand and you get the associated fitness you get the"}, {"start": 620.2800000000001, "end": 625.5200000000001, "text": " associated score of that agent in this particular environment at hand so let's"}, {"start": 625.5200000000001, "end": 632.88, "text": " now imagine that for example this agent here got a score of 150 this one got a"}, {"start": 632.88, "end": 639.72, "text": " score of let's say 60 this one got a score of like 35 this one like 40 or"}, {"start": 639.72, "end": 645.2, "text": " like I don't know like five or whatnot and you can imagine immediately see"}, {"start": 645.2, "end": 650.5600000000001, "text": " that probably this part of environment is for some reason so the weights that"}, {"start": 650.5600000000001, "end": 654.0400000000001, "text": " are in this part of the space are kind of more valuable for this particular"}, {"start": 654.0400000000001, "end": 658.76, "text": " environment and so you intuitively want to move your vector your theta vector"}, {"start": 658.76, "end": 664.36, "text": " in this direction and so how you do that practically in the in these class of"}, {"start": 664.36, "end": 667.72, "text": " evolutionary algorithms is the following so you have associated"}, {"start": 667.72, "end": 672.6800000000001, "text": " perturbation vector that you used in order to get to this particular agent"}, {"start": 672.6800000000001, "end": 679.08, "text": " from this theta initial state of set of weights so and you have like"}, {"start": 679.08, "end": 683.52, "text": " corresponding perturbation vector for all of these novel agents here in the"}, {"start": 683.52, "end": 688.88, "text": " environment and so what you'll do is a simple in practice you just do a simple"}, {"start": 688.88, "end": 693.0, "text": " weighted average of all of these vectors and you can imagine that after you do"}, {"start": 693.0, "end": 697.52, "text": " that so after you multiply this vector here with with 150 and this vector here"}, {"start": 697.52, "end": 701.84, "text": " with 60 and you do the same for these other vectors you'll end up with a"}, {"start": 701.84, "end": 708.16, "text": " resultant vector that maybe goes that maybe looks like this maybe you end up"}, {"start": 708.16, "end": 712.96, "text": " going in this direction here and so you move and because you have a learning"}, {"start": 712.96, "end": 716.96, "text": " step you'll you won't move too much you'll maybe move like here and so this"}, {"start": 716.96, "end": 720.84, "text": " is your novel theta so this is theta prime okay again re-trating what we've"}, {"start": 720.84, "end": 726.24, "text": " done here is we've we've taken the initial set of weights theta we've fit"}, {"start": 726.24, "end": 731.32, "text": " a Gaussian on top of that theta vector and what I mean by that is basically"}, {"start": 731.32, "end": 734.5600000000001, "text": " take the theta as the mean of the Gaussian and you somehow pick the"}, {"start": 734.5600000000001, "end": 738.84, "text": " covariance it's usually isotropic Gaussian then you sample n random points"}, {"start": 738.84, "end": 743.0, "text": " from that Gaussian and what you do by doing that is you sample n novel agents"}, {"start": 743.0, "end": 748.2, "text": " and then you evaluate those agents on the environment at hand you get some"}, {"start": 748.2, "end": 754.44, "text": " associated like scores and you just create this weighted sum of all of these"}, {"start": 754.44, "end": 758.6, "text": " perturbation vector and when I when I say a perturbation vector what I mean is"}, {"start": 758.6, "end": 764.24, "text": " basically this vector here so what it took to get from theta so from from"}, {"start": 764.24, "end": 772.32, "text": " from the original like vector here to get to this novel candidate agent so"}, {"start": 772.32, "end": 776.9200000000001, "text": " that's the high level understanding of what's going on now let me show you how"}, {"start": 776.9200000000001, "end": 780.4200000000001, "text": " the formulas look so that you are not confused next time when you see the"}, {"start": 780.4200000000001, "end": 784.1600000000001, "text": " formula I think it's always good to have this type of grounding and"}, {"start": 784.16, "end": 787.76, "text": " understand like visual understanding of what's going on or like just the"}, {"start": 787.76, "end": 792.8399999999999, "text": " semantics behind the notation okay let's first start and understand this this"}, {"start": 792.8399999999999, "end": 797.68, "text": " more general formula here where instead of a modeling distribution as a"}, {"start": 797.68, "end": 801.9, "text": " Gaussian you you kind of you parameterize your distribution using the"}, {"start": 801.9, "end": 806.68, "text": " the weights theta so those are the the weights you start from so this vector"}, {"start": 806.68, "end": 812.92, "text": " here and you somehow model that probability distribution and so the idea"}, {"start": 812.92, "end": 818.16, "text": " is so this this tells you the same story I just told you about so this e theta of"}, {"start": 818.16, "end": 824.1999999999999, "text": " I just means how well does this theta I agent perform on this particular"}, {"start": 824.1999999999999, "end": 828.04, "text": " environment e so this will be a score like for example hundred fifty as we saw"}, {"start": 828.04, "end": 833.92, "text": " before so let me just connect this with the drawing here so this particular set"}, {"start": 833.92, "end": 840.0, "text": " of weights here so this novel candidate vector is going to be theta I and E of"}, {"start": 840.0, "end": 844.68, "text": " theta I so that the performance of that agent on this particular environment is"}, {"start": 844.68, "end": 849.44, "text": " going to be hundred fifty as we saw here okay so that's how those two connect"}, {"start": 849.44, "end": 855.68, "text": " next up let's imagine that under the current probability distribution that's"}, {"start": 855.68, "end": 866.44, "text": " parameterized by theta this agent theta I has the probability of 0.4 and because"}, {"start": 866.44, "end": 870.48, "text": " it has such a high score like intuitively we would love to increase"}, {"start": 870.48, "end": 874.2, "text": " this probability the next time we sample from this probability distribution so"}, {"start": 874.2, "end": 880.72, "text": " optimally when we do a single step of evolutionary strategy algorithm will"}, {"start": 880.72, "end": 885.7600000000001, "text": " boost up this one to maybe 0.6 so we'll have more probability put on to on top"}, {"start": 885.7600000000001, "end": 890.6, "text": " of this particular set of weights which makes intuitively sense so again this"}, {"start": 890.6, "end": 896.9200000000001, "text": " term here tells you what's the what's the fastest way to increase the"}, {"start": 896.9200000000001, "end": 903.0, "text": " probability of this particular theta I so that's what gradient does it shows you"}, {"start": 903.0, "end": 907.84, "text": " the direction of the steepest ascent so that we can maximize this this term here"}, {"start": 907.84, "end": 912.6, "text": " and so it's weighted by the the score that that particular agent achieves and"}, {"start": 912.6, "end": 918.1600000000001, "text": " then you just do a sum across all of the end samples so and samples are are those"}, {"start": 918.16, "end": 924.76, "text": " random the random samples we take given the the the initial set of weights theta"}, {"start": 924.76, "end": 929.56, "text": " here okay that's how those two connect so that's just a more general notion of"}, {"start": 929.56, "end": 936.52, "text": " this whole evolutionary strategy setup if we assume now that the probability"}, {"start": 936.52, "end": 943.48, "text": " distribution is a Gaussian this is how this expression like transforms you can"}, {"start": 943.48, "end": 948.28, "text": " see again we have a sum across and samples this time we find how"}, {"start": 948.28, "end": 952.84, "text": " performant are these agents so this here is now going to be theta I let me just"}, {"start": 952.84, "end": 958.8000000000001, "text": " change the color so this here is a new set of weights called theta I and we"}, {"start": 958.8000000000001, "end": 964.54, "text": " just have to so epsilon subscript I is just a sample from the standard Gaussian"}, {"start": 964.54, "end": 968.5600000000001, "text": " distribution so this is basically sampling from your Gaussian and you"}, {"start": 968.56, "end": 973.76, "text": " then evaluate the that novel model on the on the environment you get the score"}, {"start": 973.76, "end": 978.56, "text": " and so as you can see here you just use those as the weights for these resultant"}, {"start": 978.56, "end": 983.8, "text": " vectors so those resultant vectors are expect are exactly these dotted lines I"}, {"start": 983.8, "end": 989.0799999999999, "text": " showed here and so yeah basically this equation is the formulaic way of"}, {"start": 989.0799999999999, "end": 993.64, "text": " describing how evolutionary search breaks okay hopefully that helped you"}, {"start": 993.64, "end": 999.24, "text": " better understand the evolutionary strategy algorithm as a final note this"}, {"start": 999.24, "end": 1008.96, "text": " theta plus sigma times epsilon sub I can be denoted as this so it's a it's a"}, {"start": 1008.96, "end": 1017.08, "text": " Gaussian with mean equal to theta and with covariance matrix equal to sigma"}, {"start": 1017.08, "end": 1021.0, "text": " identity matrix so you're basically sampling your agent from this"}, {"start": 1021.0, "end": 1026.52, "text": " distribution here you evaluate and find the score on this environment e and then"}, {"start": 1026.52, "end": 1031.16, "text": " you use that as the weight to multiply these perturbation vectors epsilon sub"}, {"start": 1031.16, "end": 1036.08, "text": " eyes okay enough ramble about evolutionary strategy let's continue and"}, {"start": 1036.08, "end": 1043.12, "text": " explain understand the poet algorithm itself okay let's see how it proceeds so"}, {"start": 1043.12, "end": 1049.96, "text": " you initialize this list of environment agents as as an empty list you add the"}, {"start": 1049.96, "end": 1054.6000000000001, "text": " initial environment and you add the initial theta so that's just gonna be a"}, {"start": 1054.6000000000001, "end": 1059.64, "text": " random vector and the initial environment is gonna be basically this"}, {"start": 1059.64, "end": 1065.44, "text": " thing here so they've been using this by by pedal Walker environment except that"}, {"start": 1065.44, "end": 1069.0, "text": " this one is already fairly complex let me show you how the initial one will"}, {"start": 1069.0, "end": 1072.3600000000001, "text": " probably look like so it will look something like this it's completely flat"}, {"start": 1072.3600000000001, "end": 1076.24, "text": " you don't even have the gap here you you will have like nothing you'll just have"}, {"start": 1076.24, "end": 1079.08, "text": " a flat environment that's gonna be the initial environment of the poet"}, {"start": 1079.08, "end": 1086.1999999999998, "text": " algorithm okay so let's go back here and then once you add that initial pair of"}, {"start": 1086.1999999999998, "end": 1092.24, "text": " environment environment agent pair so then you do basically capital T number"}, {"start": 1092.24, "end": 1096.6, "text": " of iterations and in each of those iterations you do the following so every"}, {"start": 1096.6, "end": 1103.1399999999999, "text": " now and then you are going to mutate your environments and so as you can see"}, {"start": 1103.1399999999999, "end": 1108.04, "text": " here so T is the current iteration and every now and then which is dictated by"}, {"start": 1108.04, "end": 1113.24, "text": " the end mutate you're going to execute this so what does it mean to mutate an"}, {"start": 1113.24, "end": 1119.72, "text": " environment such as the bipedal by pedal Walker environment we saw here and to"}, {"start": 1119.72, "end": 1124.32, "text": " understand that let me let me get to this table here so basically each"}, {"start": 1124.32, "end": 1130.68, "text": " environment is encoded as a as a as a vector and you can treat that vector as"}, {"start": 1130.68, "end": 1135.1599999999999, "text": " a like a gene of the environment and let's understand the format of that of"}, {"start": 1135.16, "end": 1141.76, "text": " that gene so it looks like something like this so here is our vector that"}, {"start": 1141.76, "end": 1147.8000000000002, "text": " encodes the environment and the first two coordinates here are gonna encode"}, {"start": 1147.8000000000002, "end": 1152.48, "text": " the stump the stump height as you can see here the initial values are gonna be"}, {"start": 1152.48, "end": 1158.96, "text": " like zero and then 0.4 and that's going to determine the stump heights"}, {"start": 1158.96, "end": 1164.76, "text": " throughout the environment so just to clarify what this here means is that"}, {"start": 1164.76, "end": 1171.08, "text": " your stump height so let me draw that as this so here you have like stump height"}, {"start": 1171.08, "end": 1178.04, "text": " here so stump height and you're gonna have the stump height be between 0 and"}, {"start": 1178.04, "end": 1183.56, "text": " 0.4 and you're just going to have like a uniform distribution so that means all"}, {"start": 1183.56, "end": 1188.0, "text": " of these heights here are gonna be of equal probability and throughout the the"}, {"start": 1188.0, "end": 1191.12, "text": " as the environment progresses you're gonna be sampling from this distribution"}, {"start": 1191.12, "end": 1196.6399999999999, "text": " okay next up you have gap width so the next two coordinates are gonna be gap"}, {"start": 1196.6399999999999, "end": 1200.4399999999998, "text": " width and then the next two are gonna be step width and then you're gonna have"}, {"start": 1200.4399999999998, "end": 1204.84, "text": " the step numbers and the roughness let me just kind of quickly explain the"}, {"start": 1204.84, "end": 1212.1999999999998, "text": " semantics behind these so gap width is I guess fairly clear so that's this thing"}, {"start": 1212.1999999999998, "end": 1219.84, "text": " here so this is your gap width this is the stump height then what else they"}, {"start": 1219.84, "end": 1226.72, "text": " have let me just check so we have step height and number of steps so let me let"}, {"start": 1226.72, "end": 1230.6799999999998, "text": " me see what I can find an environment with steps I think there is one here so"}, {"start": 1230.6799999999998, "end": 1234.72, "text": " you can see here so this is here we have like six steps and you can see what this"}, {"start": 1234.72, "end": 1238.9599999999998, "text": " is the step height so all of these various different parameters that"}, {"start": 1238.9599999999998, "end": 1245.1999999999998, "text": " describe the environment are encoded in this in this vector here okay so an"}, {"start": 1245.2, "end": 1250.64, "text": " important detail here is this mutation step row which tells you how are you"}, {"start": 1250.64, "end": 1254.3600000000001, "text": " going to with each duration which each mutation step how are you going to"}, {"start": 1254.3600000000001, "end": 1258.72, "text": " mutate these these parameters that describe the environment so for this"}, {"start": 1258.72, "end": 1265.48, "text": " stump height you have 0.2 which means you're going to every now and then with"}, {"start": 1265.48, "end": 1273.66, "text": " some probability boost up the 0 to like 0.2 and this 0.4 to 0.6 and now you you"}, {"start": 1273.66, "end": 1276.72, "text": " can imagine you now have a much harder environment because all of the stumps"}, {"start": 1276.72, "end": 1282.3200000000002, "text": " are going to be between like higher on average than than than before and that's"}, {"start": 1282.3200000000002, "end": 1286.88, "text": " how you're progressively and randomly mutating and making environments harder"}, {"start": 1286.88, "end": 1290.3200000000002, "text": " you can do that for all of the parameters that describe the environment"}, {"start": 1290.3200000000002, "end": 1296.4, "text": " here so all across all of these five columns dimensions cool now that we"}, {"start": 1296.4, "end": 1302.92, "text": " understand the environment let me first before getting back to the algorithm"}, {"start": 1302.92, "end": 1309.8000000000002, "text": " explain how the actual agent is constructed so you can see here how the"}, {"start": 1309.8000000000002, "end": 1314.76, "text": " agent looks like and you basically have four control signals you're controlling"}, {"start": 1314.76, "end": 1320.16, "text": " the hip torque you're controlling like so here you're controlling the knee"}, {"start": 1320.16, "end": 1325.8400000000001, "text": " torque you're controlling the other hip torque and the other knee torque"}, {"start": 1326.16, "end": 1331.2, "text": " that's the output so as for the input I think they mentioned it somewhere here"}, {"start": 1331.2, "end": 1336.92, "text": " let me just if I can zoom out they mentioned here so the agent has 10"}, {"start": 1336.92, "end": 1342.32, "text": " lighter range finders for perceiving obstacles and terrain whose measurements"}, {"start": 1342.32, "end": 1347.6000000000001, "text": " are included in the state space in other 14 state variables include hull angle"}, {"start": 1347.6000000000001, "end": 1352.3600000000001, "text": " hull angular velocity horizontal and vertical speeds positions of joins blah"}, {"start": 1352.3600000000001, "end": 1356.72, "text": " blah blah blah blah okay so all in all you have its model the following way you"}, {"start": 1356.72, "end": 1366.28, "text": " have 24 dimensional input observation or state then you have an MLP here a couple"}, {"start": 1366.28, "end": 1371.8, "text": " of hidden layers and you end up with four dimensional output vector so this"}, {"start": 1371.8, "end": 1376.48, "text": " is for this is 24 this is just an MLP and that's everything you need to know"}, {"start": 1376.48, "end": 1381.1200000000001, "text": " about the agent the final detail that's important is how do you define the"}, {"start": 1381.1200000000001, "end": 1385.64, "text": " fitness how do you find the score and here is the formulaic description of the"}, {"start": 1385.64, "end": 1392.8000000000002, "text": " reward you can see that if the robot falls then you like the robot gets minus"}, {"start": 1392.8000000000002, "end": 1397.5200000000002, "text": " hundred reward so negative reward you can see here that we are encouraging the"}, {"start": 1397.5200000000002, "end": 1405.0400000000002, "text": " robot to move like rightward so here Delta X because the X axis basically as"}, {"start": 1405.0400000000002, "end": 1409.2800000000002, "text": " you can assume goes from left to right so you want to move as much as possible"}, {"start": 1409.28, "end": 1415.96, "text": " so that's the Delta as much as possible to the right then it's penalized for the"}, {"start": 1415.96, "end": 1419.96, "text": " hull angle whereas the hull angle is this thing so basically you want to make"}, {"start": 1419.96, "end": 1425.2, "text": " sure let me zoom in a bit more so you want to make sure that this particular"}, {"start": 1425.2, "end": 1432.32, "text": " angle here so this angle here let me know it does maybe alpha you want to"}, {"start": 1432.32, "end": 1436.8, "text": " minimize it and that means you want to have this this hull being as horizontal"}, {"start": 1436.8, "end": 1442.52, "text": " as possible and the final detail they have here is they are also penalizing"}, {"start": 1442.52, "end": 1451.6399999999999, "text": " for the applied torque so if we were to explain what this means in human"}, {"start": 1451.6399999999999, "end": 1455.86, "text": " language is it's basically like a like a like a formal description of this"}, {"start": 1455.86, "end": 1462.2, "text": " informal statement please don't fall move to the right as much as possible"}, {"start": 1462.2, "end": 1467.8, "text": " try and keep your hull as horizontal as possible and try to minimize the energy"}, {"start": 1467.8, "end": 1471.96, "text": " consumption while doing all of that so that would be like an informal"}, {"start": 1471.96, "end": 1477.5, "text": " description of what this this this reward is encoding cool now that we"}, {"start": 1477.5, "end": 1483.96, "text": " understand all of that let's get back to the algorithm put main loop so we got to"}, {"start": 1483.96, "end": 1489.44, "text": " this line six here we're somehow mutating the environments we saw the"}, {"start": 1489.44, "end": 1494.16, "text": " mechanism roughly there is a pseudo algorithm in the appendix I'm going to"}, {"start": 1494.16, "end": 1499.16, "text": " show you in a minute but like for now just imagine you have like a set of"}, {"start": 1499.16, "end": 1502.56, "text": " vectors which are encoding for particular environments you can you can"}, {"start": 1502.56, "end": 1507.3200000000002, "text": " kind of imagine you mutate those vectors according to the rules we saw and then"}, {"start": 1507.3200000000002, "end": 1511.88, "text": " you get novel novel environments okay now that we have that environment a"}, {"start": 1511.88, "end": 1516.0800000000002, "text": " novel set of mutated environments we just iterate through those environments"}, {"start": 1516.08, "end": 1521.1999999999998, "text": " and we optimize the agents here as you can see in this step here we're going to"}, {"start": 1521.1999999999998, "end": 1526.08, "text": " take the the environment agent pair we're gonna do a single evolutionary"}, {"start": 1526.08, "end": 1530.6599999999999, "text": " strategy step so this as you remember produces the resultant vector so we just"}, {"start": 1530.6599999999999, "end": 1537.3999999999999, "text": " add that resultant vector to the theta that was used as the mean of the Gaussian"}, {"start": 1537.3999999999999, "end": 1541.9199999999998, "text": " and we get the novel the novel set of weights for that agent so that's how the"}, {"start": 1541.92, "end": 1547.0800000000002, "text": " agents are optimized here obviously you can see that we have like the learning"}, {"start": 1547.0800000000002, "end": 1552.76, "text": " rate alpha here we have Sigma the standard deviation for the Gaussian at"}, {"start": 1552.76, "end": 1558.72, "text": " hand and all of that combined goes into the input of this ES step and that's how"}, {"start": 1558.72, "end": 1565.16, "text": " we update the weights okay finally and I mentioned this every now and then we're"}, {"start": 1565.16, "end": 1569.8400000000001, "text": " going to so every now and then is determined by this end transfer we're"}, {"start": 1569.84, "end": 1574.04, "text": " going to evaluate candidates as you can see here we take all of the agents"}, {"start": 1574.04, "end": 1581.84, "text": " except for the theta t plus 1 M except for the Mth agent where M is the current"}, {"start": 1581.84, "end": 1587.52, "text": " agent in this loop here and we evaluate all of these agents on the Mth"}, {"start": 1587.52, "end": 1592.8, "text": " environment and we find the best performer for that environment and if"}, {"start": 1592.8, "end": 1597.6399999999999, "text": " the score that the best performer achieved is better than the than the"}, {"start": 1597.64, "end": 1603.6000000000001, "text": " current agents then the niche champion then you replace you basically overwrite"}, {"start": 1603.6000000000001, "end": 1609.44, "text": " so basically bye bye the old agent and here comes the new one which was the"}, {"start": 1609.44, "end": 1613.96, "text": " best agent which is the most adapt agent for this environment even though it came"}, {"start": 1613.96, "end": 1616.6000000000001, "text": " from some different environment so that's the this interesting cross"}, {"start": 1616.6000000000001, "end": 1620.5200000000002, "text": " pollination that's going on okay so on a high level the Poet algorithm is fairly"}, {"start": 1620.52, "end": 1627.44, "text": " simple you keep this collection of environment agent pairs you keep"}, {"start": 1627.44, "end": 1632.0, "text": " mutating the environments you keep optimizing the corresponding agents for"}, {"start": 1632.0, "end": 1636.24, "text": " those particular environments and every now and then you do this cross pollination"}, {"start": 1636.24, "end": 1641.04, "text": " step whereby for each of the environments you're going to find the"}, {"start": 1641.04, "end": 1644.72, "text": " best agent overall which may come from some different environment and replace"}, {"start": 1644.72, "end": 1650.04, "text": " the current niche champion with that one if that happens okay so that's that's"}, {"start": 1650.04, "end": 1657.12, "text": " everything and now let me just go to appendix and show you the mutation"}, {"start": 1657.12, "end": 1664.3999999999999, "text": " pseudo algorithm in a bit more detail so here we are let's see how it works so we"}, {"start": 1664.3999999999999, "end": 1668.92, "text": " have some list already we iterate through the list of agent environment"}, {"start": 1668.92, "end": 1674.44, "text": " pairs and we see if they are eligible to reproduce and what this means is they"}, {"start": 1674.44, "end": 1679.68, "text": " basically evaluate the agent M on this environment M and they see what the"}, {"start": 1679.68, "end": 1685.2, "text": " score is and if the score is like if it's not in this in this in this range"}, {"start": 1685.2, "end": 1691.44, "text": " between like for example 50 and 200 they discard that agent for that particular"}, {"start": 1691.44, "end": 1696.64, "text": " environment I mean both the agent and the environment and why is that well"}, {"start": 1696.64, "end": 1702.44, "text": " because that means that if you're outside of this range that means that"}, {"start": 1702.44, "end": 1706.0, "text": " you're either here which means you achieve super low score which means that"}, {"start": 1706.0, "end": 1710.32, "text": " the environment is probably very hard or it means that you're here which means"}, {"start": 1710.32, "end": 1715.88, "text": " it's you have a very high score and the environment is very very easy and so you"}, {"start": 1715.88, "end": 1721.2, "text": " want to kind of filter out those either too hard or too simple of environment"}, {"start": 1721.2, "end": 1725.36, "text": " after we've done that we have this parent list so basically all of the"}, {"start": 1725.36, "end": 1729.8, "text": " eligible environment agent pairs are there then we do the environment to"}, {"start": 1729.8, "end": 1733.88, "text": " produce so this step here is gonna do these random mutations you're gonna take"}, {"start": 1733.88, "end": 1736.96, "text": " the vector that describes the environment you're gonna do the"}, {"start": 1736.96, "end": 1742.0400000000002, "text": " mutations according to certain rules and you end up with this child list then you"}, {"start": 1742.0400000000002, "end": 1746.3200000000002, "text": " again just check this minimal condition IE you check whether they are in this"}, {"start": 1746.3200000000002, "end": 1751.5600000000002, "text": " range you discard the ones which are not after all of that is done there is this"}, {"start": 1751.5600000000002, "end": 1755.68, "text": " line 13 where there is something interesting going on so they rank by"}, {"start": 1755.68, "end": 1763.38, "text": " novelty and how they define novelty is using L2 metric basically they take the"}, {"start": 1763.38, "end": 1768.8400000000001, "text": " encoding vector and they compare it with all of the other environment vectors as"}, {"start": 1768.8400000000001, "end": 1773.2800000000002, "text": " well as some of the older like they have this archive of older environments and"}, {"start": 1773.2800000000002, "end": 1779.4, "text": " the further away it is from all of the other environments as defined by L2 the"}, {"start": 1779.4, "end": 1783.5200000000002, "text": " more novel it is so that's how they define the concept of novelty it's it's"}, {"start": 1783.5200000000002, "end": 1786.8000000000002, "text": " it's very different compared to the previous environments we've seen and"}, {"start": 1786.8000000000002, "end": 1793.2, "text": " they basically sort this list according to to not according to novelty and once"}, {"start": 1793.2, "end": 1798.0, "text": " we've done all this they also here they have this evaluation of candidates so"}, {"start": 1798.0, "end": 1803.04, "text": " they find the best agent for that particular environment and they and if"}, {"start": 1803.04, "end": 1809.6000000000001, "text": " this minimal condition is satisfied then they add the environment agent pair to"}, {"start": 1809.6000000000001, "end": 1813.92, "text": " the list so what can potentially happen here again is that some of the other"}, {"start": 1813.92, "end": 1820.44, "text": " agents is better at the environment that can this at this child environment and"}, {"start": 1820.44, "end": 1826.96, "text": " then you're just gonna overwrite the agent with this better agent here so"}, {"start": 1826.96, "end": 1831.64, "text": " yeah there is a it's kind of intricate but you got the gist of it already you"}, {"start": 1831.64, "end": 1836.0800000000002, "text": " have before I showed you this this pseudo algorithm okay let's get back to"}, {"start": 1836.0800000000002, "end": 1843.44, "text": " the main paper and see where we stopped okay let's get to the results the first"}, {"start": 1843.44, "end": 1848.8, "text": " comparison they do is they compare poet with the evolutionary strategy from"}, {"start": 1848.8, "end": 1853.84, "text": " scratch which means you find some fairly complicated environment as the one you"}, {"start": 1853.84, "end": 1860.8, "text": " can see here and you either train the agent using poet or you directly train"}, {"start": 1860.8, "end": 1864.6, "text": " evolutionary search so you start from a random agent and you try by random"}, {"start": 1864.6, "end": 1868.36, "text": " perturbations to find the agent that performs well for this particular"}, {"start": 1868.36, "end": 1872.86, "text": " environment and it turns out that as soon as the environment gets like"}, {"start": 1872.86, "end": 1877.44, "text": " decently complicated the evolutionary strategy from scratch starts failing and"}, {"start": 1877.44, "end": 1881.96, "text": " they mention here some of the failure cases so they say here in effect to"}, {"start": 1881.96, "end": 1886.52, "text": " obtain positive scores these agents learn to move forward but also to freeze"}, {"start": 1886.52, "end": 1890.56, "text": " before challenging obstacles which helped them avoid the penalty of minus"}, {"start": 1890.56, "end": 1895.4, "text": " hundred for falling down and this behavior is a local optimum the agents"}, {"start": 1895.4, "end": 1899.4, "text": " could in principle learn to overcome the obstacles but instead converge on"}, {"start": 1899.4, "end": 1904.2, "text": " playing it safe by not moving it's a very interesting behavior like a like a"}, {"start": 1904.2, "end": 1910.6000000000001, "text": " like a risk adversity like being manifested here in these simple agents"}, {"start": 1910.6000000000001, "end": 1915.24, "text": " in the most simple of environments and yeah I guess we see that in the in the"}, {"start": 1915.24, "end": 1920.52, "text": " real happening in the real world as well so here you can see the results you can"}, {"start": 1920.52, "end": 1925.6000000000001, "text": " see that that that the ES kind of gets stuck here and stays in the same place"}, {"start": 1925.6000000000001, "end": 1929.6000000000001, "text": " because otherwise it will fall down and get a minus hundred reward which is"}, {"start": 1929.6, "end": 1934.9199999999998, "text": " suboptimal whereas poet can actually yeah can actually kind of jump across"}, {"start": 1934.9199999999998, "end": 1940.1599999999999, "text": " these these gaps and and and continue on doing its job similarly for these other"}, {"start": 1940.1599999999999, "end": 1945.36, "text": " types of environments you can see here as soon as a big obstacle comes here the"}, {"start": 1945.36, "end": 1951.32, "text": " agent gets stuck whereas the poet keeps on going to the right same here it just"}, {"start": 1951.32, "end": 1955.28, "text": " stops in front of a big obstacle and does not move and that's the local"}, {"start": 1955.28, "end": 1962.28, "text": " optimum that they were referring to okay let's continue here you can see that"}, {"start": 1962.28, "end": 1967.44, "text": " poet can solve even some of very difficult environments and that's that's"}, {"start": 1967.44, "end": 1972.76, "text": " very very cool results okay here is just the quantitative visualization of how"}, {"start": 1972.76, "end": 1978.08, "text": " like of this huge gap between poet performance and evolutionary search by"}, {"start": 1978.08, "end": 1981.2, "text": " the way I'm butchering the names here like I'm not sure whether you can use"}, {"start": 1981.2, "end": 1987.0, "text": " interchangeably strategy search and like just evolutionary system because I think"}, {"start": 1987.0, "end": 1991.32, "text": " opening I had some evolutionary strategy paper so it may be the name may be"}, {"start": 1991.32, "end": 1997.4, "text": " overloaded but I explained the semantics between behind the evolutionary search"}, {"start": 1997.4, "end": 2003.0800000000002, "text": " like a couple minutes ago so hopefully that won't be confusing okay now we've"}, {"start": 2003.0800000000002, "end": 2008.88, "text": " seen the direct comparison between poet and evolutionary search now you can do"}, {"start": 2008.88, "end": 2015.1200000000001, "text": " something a bit more smart and that's doing a curriculum based learning but"}, {"start": 2015.1200000000001, "end": 2019.68, "text": " like not using poet which is highly open-ended but doing a more direct path"}, {"start": 2019.68, "end": 2023.88, "text": " curriculum learning let's see what what that means so a natural question then is"}, {"start": 2023.88, "end": 2027.7600000000002, "text": " whether the environments created and solved by poet can also be solved by an"}, {"start": 2027.7600000000002, "end": 2033.16, "text": " explicit direct path curriculum building control algorithm the sequence of"}, {"start": 2033.16, "end": 2036.48, "text": " environment starts with an environment of only flat ground without any"}, {"start": 2036.48, "end": 2040.2, "text": " obstacles which is easy enough for any randomly initialized agent to quickly"}, {"start": 2040.2, "end": 2044.64, "text": " learn to complete then each of the subsequent environments are constructed"}, {"start": 2044.64, "end": 2048.64, "text": " by slightly modifying the current environment okay let me try and explain"}, {"start": 2048.64, "end": 2053.52, "text": " what that means and what's the difference in approach so on one hand"}, {"start": 2053.52, "end": 2058.88, "text": " you have poet so you start with some random environment you start with the"}, {"start": 2058.88, "end": 2062.0, "text": " easy environment and then you randomly start protruding that environment so"}, {"start": 2062.0, "end": 2064.84, "text": " it's environment number one you do some mutations you get to environment number"}, {"start": 2064.84, "end": 2070.1200000000003, "text": " two and then you do some like mutations you get to three and then you get to"}, {"start": 2070.1200000000003, "end": 2075.6400000000003, "text": " four and then etc etc and you end up with environment after n steps which is"}, {"start": 2075.6400000000003, "end": 2080.76, "text": " somewhere here okay so now what they propose is let's start from this very"}, {"start": 2080.76, "end": 2086.2000000000003, "text": " same so so this is the original environment which is the flat ground"}, {"start": 2086.2000000000003, "end": 2089.88, "text": " everything super simple let me just duplicate it here for the sake of easier"}, {"start": 2089.88, "end": 2093.52, "text": " visualization so we have the same environment so these two here are the"}, {"start": 2093.52, "end": 2097.0, "text": " same so that was that's the simplest environment possible now instead of"}, {"start": 2097.0, "end": 2102.08, "text": " having this highly nonlinear mutation path towards M why not just go directly"}, {"start": 2102.08, "end": 2108.8, "text": " towards N and when I say directly as you know the environment described by a"}, {"start": 2108.8, "end": 2113.12, "text": " vector and you can literally linearly interpolate between these two vectors"}, {"start": 2113.12, "end": 2118.8, "text": " and create a set of environments that goes from one to N but following a much"}, {"start": 2118.8, "end": 2123.12, "text": " more direct path and they mentioned they explained here how exactly they do that"}, {"start": 2123.12, "end": 2127.44, "text": " they say more specifically to get a new environment each obstacle parameter of"}, {"start": 2127.44, "end": 2131.52, "text": " the current environment has an equal chance of staying the same value or"}, {"start": 2131.52, "end": 2135.92, "text": " increasing by corresponding mutation step value in table one until that"}, {"start": 2135.92, "end": 2141.12, "text": " obstacle parameter reaches that of the target environment so they do introduce"}, {"start": 2141.12, "end": 2147.0, "text": " a slight stochasticity in this process is not just a pure linear interpolation"}, {"start": 2147.0, "end": 2151.72, "text": " but like it's very close to that okay so this is the approach they take and let's"}, {"start": 2151.72, "end": 2159.16, "text": " now see the results they got comparing to this baseline okay here are the"}, {"start": 2159.16, "end": 2165.12, "text": " results these are something called rose plots and you can see that so run one is"}, {"start": 2165.12, "end": 2170.0, "text": " a single run of the poet algorithm and as poet was was running you can imagine"}, {"start": 2170.0, "end": 2175.04, "text": " that the environments were getting harder and harder let me first explain"}, {"start": 2175.04, "end": 2180.64, "text": " to you how you can parse this this this diagram so each of these five axes here"}, {"start": 2180.64, "end": 2185.04, "text": " correspond to one of the five parameters we saw in the table before so that means"}, {"start": 2185.04, "end": 2192.64, "text": " one is coding for like gap one is coding for the stump size one is coding for the"}, {"start": 2192.64, "end": 2198.04, "text": " roughness etc etc so the hardest environments are the ones which would be"}, {"start": 2198.04, "end": 2203.14, "text": " touching basically the end points here so these would be the this would be the"}, {"start": 2203.14, "end": 2207.92, "text": " hardest environment possible so this one here this is the hardest environment"}, {"start": 2207.92, "end": 2213.96, "text": " okay so looking at these these plots here we can see that so we have here"}, {"start": 2213.96, "end": 2217.28, "text": " three independent fronts let me first explain what the columns mean and what"}, {"start": 2217.28, "end": 2222.64, "text": " the what the rows mean so we have three independent runs as and the run like the"}, {"start": 2222.64, "end": 2226.04, "text": " the environments were getting harder and harder as the as the runs were"}, {"start": 2226.04, "end": 2233.16, "text": " progressing and then we can notice that the blue pentagons is the direct path"}, {"start": 2233.16, "end": 2239.12, "text": " curriculum and we can see that it's not able to to achieve the same to solve"}, {"start": 2239.12, "end": 2245.2, "text": " the same complexity level environments as poet given the similar amount of"}, {"start": 2245.2, "end": 2251.12, "text": " compute so again connecting this diagram with the thing I draw here"}, {"start": 2251.12, "end": 2257.24, "text": " basically the the the poet does not achieve never gets to end it maybe gets"}, {"start": 2257.24, "end": 2262.2, "text": " to like manages to solve like some of the environments the more the more these"}, {"start": 2262.2, "end": 2265.52, "text": " easier environments but never manages to actually solve these harder"}, {"start": 2265.52, "end": 2269.7999999999997, "text": " environments that's what we can see visually on this on these rose plots"}, {"start": 2269.7999999999997, "end": 2277.6, "text": " here continuing on this is just another representation of that same fact and one"}, {"start": 2277.6, "end": 2281.8799999999997, "text": " thing that's worth mentioning about poet is the following so while poet cannot"}, {"start": 2281.8799999999997, "end": 2286.2799999999997, "text": " guarantee reaching a particular preconceived target these results"}, {"start": 2286.2799999999997, "end": 2290.12, "text": " suggested in some environment spaces poets unique ability to generate"}, {"start": 2290.12, "end": 2294.7599999999998, "text": " multiple different challenges and solve them may actually still provide a more"}, {"start": 2294.7599999999998, "end": 2299.2, "text": " promising path towards solving some preconceived target challenges so what"}, {"start": 2299.2, "end": 2304.52, "text": " this sentence here is telling us is the following so the thing with poet is you"}, {"start": 2304.52, "end": 2309.64, "text": " can solve arbitrarily complicated environments but like it's not like you"}, {"start": 2309.64, "end": 2315.04, "text": " can specify like up in front like you cannot upfront specify this particular"}, {"start": 2315.04, "end": 2319.12, "text": " environment and tell poet hey go and solve this environment which I deeply"}, {"start": 2319.12, "end": 2323.2799999999997, "text": " care about no you can't do that so that's that's that's what this approach"}, {"start": 2323.2799999999997, "end": 2328.68, "text": " here try to do and we show that it failed so the whole point is you have to"}, {"start": 2328.68, "end": 2332.64, "text": " have this open-ended process and you have to be collecting milestones and"}, {"start": 2332.64, "end": 2336.68, "text": " maybe some of the milestones turn out to be very useful for the problem you care"}, {"start": 2336.68, "end": 2341.88, "text": " about so Kenneth Stanley one of the co-authors of this paper built like a"}, {"start": 2341.88, "end": 2346.52, "text": " whole philosophy behind this this this discovery here he even has a book why"}, {"start": 2346.52, "end": 2353.24, "text": " greatness cannot be planned where he is discussing this this tension that"}, {"start": 2353.24, "end": 2357.44, "text": " exists between having clear objectives and where you want to get to versus"}, {"start": 2357.44, "end": 2364.0, "text": " having this more divergent approach and he's arguing very very very very"}, {"start": 2364.0, "end": 2368.18, "text": " strongly for this more open-ended approach as opposed to these direct"}, {"start": 2368.18, "end": 2374.84, "text": " optimization methods and we saw like and he uses results such as this one to to"}, {"start": 2374.84, "end": 2380.0, "text": " argument why that might be the best thing to do in the long run and that's"}, {"start": 2380.0, "end": 2383.6000000000004, "text": " something that evolution itself is doing it's not like evolution is going towards"}, {"start": 2383.6000000000004, "end": 2391.52, "text": " a particular solution okay let me explain you this very very interesting"}, {"start": 2391.52, "end": 2396.8, "text": " diagram that that kind of shows the power of the poet algorithm so here you"}, {"start": 2396.8, "end": 2400.82, "text": " can see the parent environment and this particular agent at iteration 400"}, {"start": 2400.82, "end": 2406.28, "text": " learned to do this type of like it's dragging its leg very close to the"}, {"start": 2406.28, "end": 2410.36, "text": " ground so it's a very we can see it's a very suboptimal behavior so the agent"}, {"start": 2410.36, "end": 2415.84, "text": " ended up in some local optimum and it cannot get unstuck so what poet does is"}, {"start": 2415.84, "end": 2422.32, "text": " at one point of time this agent is gonna perform outperform some other agent on"}, {"start": 2422.32, "end": 2426.04, "text": " some other environment so that's the transfer we saw multiple times and you"}, {"start": 2426.04, "end": 2431.16, "text": " see that in this environment there are stumps and after training on this so"}, {"start": 2431.16, "end": 2434.56, "text": " after training this same agent so the same set of weights on this novel"}, {"start": 2434.56, "end": 2439.64, "text": " environment the agent learned how to straighten up the legs you can see after"}, {"start": 2439.64, "end": 2446.24, "text": " iteration 1175 the agent is already walking quite like the legs are"}, {"start": 2446.24, "end": 2451.88, "text": " straightened out and then what happens is after you now copy that agent into"}, {"start": 2451.88, "end": 2456.56, "text": " the old environment and you keep on iterating you can see that here in the"}, {"start": 2456.56, "end": 2460.48, "text": " parent environment the agent finally learned how to walk straight so if you"}, {"start": 2460.48, "end": 2465.52, "text": " did not have this this this dynamics where the agent was sent to another"}, {"start": 2465.52, "end": 2468.36, "text": " environment trained there and then returned back to this environment you"}, {"start": 2468.36, "end": 2473.6, "text": " would never get unstuck from this local optimum we see here and so that's that's"}, {"start": 2473.6, "end": 2478.0, "text": " the whole that's a very powerful thing about poet I guess the main the main"}, {"start": 2478.0, "end": 2483.28, "text": " con side of poet is you cannot as we saw we cannot directly solve the"}, {"start": 2483.28, "end": 2486.72, "text": " environments we care about we kind of have to be collecting milestones and"}, {"start": 2486.72, "end": 2490.88, "text": " hope for what and hope that some of the milestones make sense for what we care"}, {"start": 2490.88, "end": 2495.12, "text": " about okay let me wrap up this paper by stating the possum some possible"}, {"start": 2495.12, "end": 2501.08, "text": " extensions of poet although obviously the paper was published in 2019 so I'm"}, {"start": 2501.08, "end": 2506.48, "text": " not quite sure whether any of these extensions happened or did not okay so"}, {"start": 2506.48, "end": 2510.32, "text": " they say the following an additional constraint that exists in this initial"}, {"start": 2510.32, "end": 2515.36, "text": " work is that the body of the agent is fixed ultimately limiting the sort of"}, {"start": 2515.36, "end": 2521.2400000000002, "text": " obstacles it can overcome for example how big of a gap it can jump over here"}, {"start": 2521.2400000000002, "end": 2525.88, "text": " too a more powerful and expressive encoding of morphologies including when"}, {"start": 2525.88, "end": 2530.52, "text": " I say morphologies they mean body shapes including those like CPP ends"}, {"start": 2530.52, "end": 2535.72, "text": " they're based on developmental biology could allow us to co-evolve the"}, {"start": 2535.72, "end": 2540.16, "text": " morphology and the body of the agent along with its brain in addition to the"}, {"start": 2540.16, "end": 2544.3599999999997, "text": " environment it is solving I mean this is a very straightforward extension of the"}, {"start": 2544.3599999999997, "end": 2549.8799999999997, "text": " of the research that was presented in this paper you can imagine that not only"}, {"start": 2549.8799999999997, "end": 2556.9199999999996, "text": " can you mutate the environment and then adapt the agents to that environment but"}, {"start": 2556.9199999999996, "end": 2561.2, "text": " you can imagine that you can also have a encoding vector for the morphology of"}, {"start": 2561.2, "end": 2566.3999999999996, "text": " the agent and you could be mutating the more that encoding vector and by doing"}, {"start": 2566.3999999999996, "end": 2569.9199999999996, "text": " those mutations you can maybe end up with an agent that has like longer legs"}, {"start": 2569.9199999999996, "end": 2575.8399999999997, "text": " or you can also imagine that you could be mutating the actual like brain"}, {"start": 2575.8399999999997, "end": 2580.4399999999996, "text": " architecture of the agent so that means instead of instead of like having that"}, {"start": 2580.4399999999996, "end": 2587.72, "text": " fixed MLP I described a couple of minutes ago so something like this you"}, {"start": 2587.72, "end": 2591.52, "text": " can be modifying this architecture as well mutating the architecture so you're"}, {"start": 2591.52, "end": 2595.9599999999996, "text": " mutating the architecture of the brain of the body and you can also imagine you"}, {"start": 2595.9599999999996, "end": 2600.52, "text": " could be mutating the rewards for each of the environments so not why just keep"}, {"start": 2600.52, "end": 2606.48, "text": " the why keep the reward constant as we saw there like with minus hundred"}, {"start": 2606.48, "end": 2611.7999999999997, "text": " penalty for for falling etc etc let me just find this is a super long paper so"}, {"start": 2611.7999999999997, "end": 2617.22, "text": " you could be also mutating the reward and all of that additional diversity"}, {"start": 2617.22, "end": 2622.7599999999998, "text": " could yeah probably either end up going towards some space that we don't care"}, {"start": 2622.7599999999998, "end": 2627.12, "text": " about or it could be very interesting although you probably would not want to"}, {"start": 2627.12, "end": 2632.08, "text": " give the reward too much flexibility you want to kind of scope it you roughly"}, {"start": 2632.08, "end": 2635.4399999999996, "text": " know what you care about and then you form a space you maybe form a"}, {"start": 2635.4399999999996, "end": 2640.3999999999996, "text": " probability distribution across rewards so that some rewards are more likely"}, {"start": 2640.3999999999996, "end": 2645.24, "text": " than some other ones okay those were some candies for your thought hopefully"}, {"start": 2645.24, "end": 2651.24, "text": " you liked this this video hopefully found this this idea of poet an open"}, {"start": 2651.24, "end": 2655.72, "text": " endedness interesting and if you did share the video out consider"}, {"start": 2655.72, "end": 2675.64, "text": " subscribing join the discord community and until next time bye bye"}]