DeepMind's Android RL Environment - AndroidEnv
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I give you some background details behind the newly introduced AndroidEnv and I show you what you need to modify in order to use it for an arbitrary Android app.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

✅ blog: https://deepmind.com/research/publications/androidenv
✅ GitHub: https://github.com/deepmind/android_env
✅ paper/technical report: https://arxiv.org/abs/2105.13231

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Introducing AndroidEnv - why is it important?
03:00 Tech report intro
05:00 Real-time simulation - novel problems for RL agents
07:10 Raw action space, gestures, etc.
11:30 Generic RL agents - results on a selected set of tasks
13:45 Creating a task for your custom Android app
16:31 A set of existing task examples

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#androidenv #deepmind #reinforcementlearning
 Introducing Android M, an open-ended platform for training agents on Android apps and games. With a universal touchscreen interface, access to the entire operating system, and a number of ready-to-use tasks, Android M is a promising domain for RL research. So I just stumbled across this tweet a couple of days ago, and I thought covering this because it seems like a really, really impactful future environment. So I'm going to walk you through their blog. So they have a blog here, and they also open-sourced a GitHub repo with instructions how to get this started and start training your RL agents on Android, different Android applications. So they also have a technical report. I'll walk you through that one as well, just some important details. And by the end of this video, hopefully you'll understand why this is important and how you can get started playing with your own Android RL agents. So first of all, probably the most obvious thing is this. So they say here, the increasing complexity of environments has driven the development of novel algorithms and agents such as TQN. So that's the Deep Q network from 2013, and Atari learning environment, ALE, was the thing that basically made it happen aside from convolutional neural networks. But yeah, just focusing on environments. Then we had AlphaGo, again, Go simulator, PPO, like OpenAI's algorithm, proximal policy optimization. They used Mojoco to develop that one, and again, AlphaStar used StarCraft II environment. So in order to advance the state of the art, even further, researchers seek new and more stimulating environments to tackle. So the next step, and this is really exciting, basically how you can take whatever Android application and train an RL agent with a couple of caveats. So basically, and we'll see that in the tech report and on the GitHub later, but you'll need to have an open source app, obviously, and you'll need to modify the source code a bit so that it outputs the rewards, which then you'll parse via custom-made tasks, and finally you'll have the complete thing. So you know, you'll define when the episode ends, what are the rewards in your environment, and then you can train the agents. But other than that, it's really cool, and yeah, let's continue here. Yeah, you can see some examples of a contact app, and they're using here, like whatever, like opening Play Store, and you can even use YouTube, you can browse the web, you can do basically whatever, you can play games, you can do whatever Android offers. The whole ecosystem of these apps is now basically open for developing custom reinforcement learning agents, and you can see here some games like 2048, etc. So let me just jump straight into the tech report, and then we'll see how you can actually create those custom tasks so you can train your own RL agents. So here is the paper, and I just want to walk you through the main ideas here. So first things first, so this is the obvious one, but screen pixels constitute the observations, the action space is defined by touchscreen gestures, and this is not completely true, we'll see what the raw action space looks like in a minute. The interaction is real-time, and this is really important, and the actions are executed asynchronously while the environment runs at its own time scale. So this means that basically the environment is not a lockstep environment like Atari, where you basically send an action, so the environment waits until you figure out what your next action will be, and then once you send the action to the environment, it executes a step forward, and then waits again for your next action. So that's the lockstep paradigm. Here on the other hand, it's a real-time environment, that means Android emulator doesn't care whether you skip certain actions if you were too slow to compute that action or whatever was the problem. So that's something that makes this even more challenging than those other simulators. And we'll dig in a bit more details in a couple of minutes there as well, but I just want to state this one. So the sheer number of apps built for a multitude of important aspects of human life ranging from education and business to communication and entertainment provides virtually unlimited challenges for RL research. So this is going to be big in my opinion. We now have a huge taskpad of applications which are actually meaningful, like comparing that to Atari games or those made-up environments. It will be so much easier to actually, after training these agents, to deploy them into some product, so there is a much lesser gap between this environment and the thing you actually want and between compared to Atari or some other environments. Okay, so let's continue here. So I already mentioned, so Android M is unable to run in lockstep. And let me jump over all of these and first tackle that problem. Okay, so here are some additional details. So they say here, another important factor to consider in real-time environments is that agents require some deliberation time to generate the next action given an observation. So that means if you have an agent that's a convolutional neural network, once it receives the observation, it will take some time to process that observation in order to output the action. And that doesn't matter in those lockstep environments, but here the timings actually do matter. So they say it here, in a traditional lockstep environment, the environment generates an observation and pauses to wait until the agent responds with an action before stepping the simulation forward. So, okay, and the diagram is here and basically, so once you get the observation, they kind of compress the deliberation time because it doesn't matter. So basically, once you compute it, you issue an action and then the environment steps forward and then just repeat. And you can see there are no problems here. But on the other hand, if you have a real-time environment like Atari, like Android M, the large deliberation time can be harmful to performance. And they have a nice chart here as well. So here the action sends the, so the agent sends the action and you can basically, in order to cope with this problem of this being a real-time environment, you can insert these artificial weights inside your routine. And so what happens is once you issue an action, the simulator starts rendering the observation. So this may vary depending on the app, et cetera. And then once you get the observation, it takes some time for the agent to compute the next action and then it issues the action. And as you can see, because of the variance of this thing and of the rendering, you'll have to kind of think about this a bit more than in your lockstep environments. So that's something to keep in mind. It's going to make this a bit more challenging, but yeah, I guess it's not that big of a deal. Okay. Let me get back to the beginning of the stack report and let's start with the action interface. So basically the action interface is super simple. So the native action space of the environment consists of a tuple consisting of a position. Basically you have X and Y coordinates which are in the continuous space. And then the Android M will, depending on the actual device you're using, the resolution of your emulator, it will just discretize these continuous actions. Continuous actions into those pixels. So that is, it's basically, you don't have to care about the actual resolution. The second part of your action will be this discrete value. So it can be touch, lift, or just repeat. Basically, yeah, you can either touch the screen, you can either lift the pointer and you can repeat the previous action. So that's the raw action space. And the thing is in order to actually, in order to meaningful interact with Android applications, as you may know, you need these gestures. So which are much more complex. And they say it here, the complexity of the interface arises from the fact that individual raw actions on their own do not necessarily trigger a meaningful change in the environment. And they have nice visualizations here. Basically you can see the swiping and you can see simple touch and lift. You can see a bit more like nonlinear swipe movement. And basically, this makes it so much harder for the agent to learn because you have to have, you have to be both precise in the spatial sense and also in the time sense. So you can't do it like this and then stop here and then continue doing the swipe because the system, the Android itself, probably won't interpret that as a correct gesture as a swipe action. So you'll have, so the agent will have to both be precise both time-wise and space-wise. Let's call it that way. And I think they mention it here. So this need to compose the actions pair with the difficulty of solving the underlying task itself leads to difficult exploration problem. So for example, in order to learn to play chess, an agent must not only find a winning strategy, it also has to learn to move pieces through drag and drop gestures. So yeah, that's the part I mentioned. And that's something to keep in mind. Aside from these information from the observation, which is basically an RGB frame, you'll have additional text task extras. And so an extra in Android Envy is any information that the environment sends to aid the understanding of the task. The information sent through this channel is typically very useful for learning, yet difficult to extract from raw pixels. And I gave an example of, I think, like text being displayed on the screen. And instead of you having to parse and learn the OCR additionally, you'll have the option of just circumventing that and printing that, like just putting that into this extras variable, which will contain the string directly, making it a bit easier to learn the task at hand. So I mentioned that in order to actually deploy your RL agent to these applications, you'll have to modify the apps in a way in order to create the notion of reward and of episode signal, et cetera. So they say here, while Android is an operating system with no inherent rewards or episodes, Android Env provides a simple mechanism for defining tasks on it. And we'll see how that functions. I'll just walk you through the GitHub and that's going to be the best explanation, I guess. They made a small selection of tasks already, and you can see some apps like Catch here and RocketSlave, PressButton, some apps like 2048, like this is a popular game, and Blockinger. And you can see the layout here is pretty complex. And they say it here, for example, most agents perform well on tasks such as Catch. So this is a simple one. They have a simple action interface and dense rewards, whereas the combination of a highly structured interface, you can see the complex layout here, like all these buttons and here, the actual screen of the game. Time sensitivity and sparse rewards renders Blockinger particularly difficult to solve. And they tested a bunch of common RL agents. And the interesting thing was that actually DQN, one of the oldest agents out there, like deep learning agents at least, deep RL agents, has maybe one of the best performances on all of these tasks. And as you can see, these cyan curves, DQN outperforms many of these other agents like RTD2 or Impala, etc. And you can also see the progression, like going from top left here to bottom right, you can see that the tasks get harder and harder, and basically the performance really drops down. And yeah. Okay, by the way, for those of you who haven't been developing Android apps, let me just kind of give you some intuition here. Basically, when you open up an Android studio, and you can basically in order to debug your app, you initialize this emulator, which can be arbitrary device, which you need to specify, and it will have certain height and certain width of the screen. So you can also pick a specific version of Android. So all of that can be specified through Android Studio. So these are just some things you won't have to worry about because the emulators themselves care about this part. And that's just abstracted from you. Additionally, they mentioned a bunch of useful environments that were previously open sourced or advised. Some of them are like you have to pay the license like Mojoco. But they end up here saying that since Android has billions of users and Android M provides tasks that run on the standard Android operating system simulator, agents trained on the platform could potentially tackle a wide range of use cases leading to direct real world impact. And this is something I'm really passionate about. So seeing these agents and RL in general, having some real world impact will be super, super interesting. And I hope to see like research and apps happening because of this new environment. So having said that, let me go back to the GitHub and show you how to actually create a task. Okay, so if you open up this tasks guide MD file, you'll see these two sections here where they basically describe what you need to do in order to enable your Android application to be used as a training ground for your your RL agent. And the main things are so you have this setup steps, which will once the once the training start, it will just do a couple of things like you can automate installation of the the application from the APK APK file, which is just Android binary. You can do a bunch of those stuff. And I'll just focus on the stuff important to RL. And that's, that's defining rewards and like end of episode signal. So you'll basically have to have to set up this log parsing config in order to just define certain regs expressions. Because the way that apps will communicate the reward information, as I think I mentioned, is through the log log, basically through the log console. So here's an example of a text proto file, which defines the task. And if I expand it, you can see so it's pretty self explanatory. So perform these upon launching the environment, install the app, check if it's installed correctly, etc. So that's the not that interesting part. Let me just skip down to the this part. And basically, you can see you define the regx expression for your reward and for your score for the episode and signal etc. And the way you know this all works is the following. So you might you might have noticed the tasks often rely on log messages exposed by the Android system, which Android app can intercept and translate into items such as rewards, episode and signals or task extras. And they said here, so of course, applications might not send suitable messages by default. So in order to have access to such messages, we often add them to the app source code to match our expectation. For example, in the case of the 2048 app, we find in the game source code the exact lines where the score is computed, and add a line to log this value in the format that is expected by the text proto. So that's this file here. So as you can see here, some regular expressions, we have a plus or minus sign, then we have a couple of digits followed by a dot and then a couple of digits more. So basically a float like a scalar value. And so you add the line to log this value in the format that is expected by the text proto, or conversely, make sure that yeah, whatever. So basically, you'll have to edit the source code of your application by adding a logging line here. And then through this text proto, you'll be able to send that environment to the agent. And that's how you're going to actually train and set up everything. And finally, they have a couple of examples here on this page, example tasks. And I'll just link all of these in the description. But basically, here is a really simple environment they created. And in this one, so take like, take this one as an example. So basically, the RL agent gets a reward plus one, if it clicks the B button, and if it clicks the A button, it just the episode ends, because that's how they defined it. And basically, what happens then is the app shuts down, and the environment will automatically launch relaunch the app again. And that's done through all through the task proto file we just saw. So that's how it works. And there are, aside from this one, they have a bunch of other examples, like with a calculator, some games, like the 2084 dimension, for which they had to modify the source code, and like chess. So yeah, bunch of different stuff. And yeah, hopefully you found this one useful. If you did, leave a like, subscribe and see you in the next video.
[{"start": 0.0, "end": 5.5200000000000005, "text": " Introducing Android M, an open-ended platform for training agents on Android apps and games."}, {"start": 5.5200000000000005, "end": 9.44, "text": " With a universal touchscreen interface, access to the entire operating system,"}, {"start": 9.44, "end": 14.96, "text": " and a number of ready-to-use tasks, Android M is a promising domain for RL research."}, {"start": 14.96, "end": 20.240000000000002, "text": " So I just stumbled across this tweet a couple of days ago, and I thought covering this because"}, {"start": 20.240000000000002, "end": 27.84, "text": " it seems like a really, really impactful future environment. So I'm going to walk you through"}, {"start": 27.84, "end": 33.68, "text": " their blog. So they have a blog here, and they also open-sourced a GitHub repo"}, {"start": 34.4, "end": 39.68, "text": " with instructions how to get this started and start training your RL agents on Android,"}, {"start": 40.32, "end": 44.879999999999995, "text": " different Android applications. So they also have a technical report. I'll walk you through"}, {"start": 44.879999999999995, "end": 50.0, "text": " that one as well, just some important details. And by the end of this video, hopefully you'll"}, {"start": 50.0, "end": 57.120000000000005, "text": " understand why this is important and how you can get started playing with your own Android RL agents."}, {"start": 57.12, "end": 62.879999999999995, "text": " So first of all, probably the most obvious thing is this. So they say here,"}, {"start": 62.879999999999995, "end": 67.2, "text": " the increasing complexity of environments has driven the development of novel algorithms and"}, {"start": 67.2, "end": 76.56, "text": " agents such as TQN. So that's the Deep Q network from 2013, and Atari learning environment,"}, {"start": 76.56, "end": 82.0, "text": " ALE, was the thing that basically made it happen aside from convolutional neural networks."}, {"start": 82.0, "end": 87.76, "text": " But yeah, just focusing on environments. Then we had AlphaGo, again, Go simulator, PPO,"}, {"start": 87.76, "end": 95.28, "text": " like OpenAI's algorithm, proximal policy optimization. They used Mojoco to develop that"}, {"start": 95.28, "end": 100.48, "text": " one, and again, AlphaStar used StarCraft II environment. So in order to advance the state"}, {"start": 100.48, "end": 105.28, "text": " of the art, even further, researchers seek new and more stimulating environments to tackle."}, {"start": 105.28, "end": 110.56, "text": " So the next step, and this is really exciting, basically how you can take whatever Android"}, {"start": 110.56, "end": 116.72, "text": " application and train an RL agent with a couple of caveats. So basically, and we'll see that in"}, {"start": 116.72, "end": 121.92, "text": " the tech report and on the GitHub later, but you'll need to have an open source app, obviously,"}, {"start": 121.92, "end": 127.28, "text": " and you'll need to modify the source code a bit so that it outputs the rewards, which then you'll"}, {"start": 127.28, "end": 134.24, "text": " parse via custom-made tasks, and finally you'll have the complete thing. So you know, you'll define"}, {"start": 134.24, "end": 139.44, "text": " when the episode ends, what are the rewards in your environment, and then you can train the agents."}, {"start": 139.44, "end": 145.44, "text": " But other than that, it's really cool, and yeah, let's continue here. Yeah, you can see some"}, {"start": 145.44, "end": 151.44, "text": " examples of a contact app, and they're using here, like whatever, like opening Play Store,"}, {"start": 152.0, "end": 159.12, "text": " and you can even use YouTube, you can browse the web, you can do basically whatever, you can play"}, {"start": 159.12, "end": 166.64, "text": " games, you can do whatever Android offers. The whole ecosystem of these apps is now basically"}, {"start": 166.64, "end": 173.11999999999998, "text": " open for developing custom reinforcement learning agents, and you can see here some games like 2048,"}, {"start": 173.11999999999998, "end": 178.0, "text": " etc. So let me just jump straight into the tech report, and then we'll see how you can actually"}, {"start": 178.0, "end": 183.76, "text": " create those custom tasks so you can train your own RL agents. So here is the paper, and I just"}, {"start": 183.76, "end": 188.88, "text": " want to walk you through the main ideas here. So first things first, so this is the obvious one,"}, {"start": 188.88, "end": 193.67999999999998, "text": " but screen pixels constitute the observations, the action space is defined by touchscreen gestures,"}, {"start": 193.68, "end": 198.48000000000002, "text": " and this is not completely true, we'll see what the raw action space looks like in a minute."}, {"start": 199.52, "end": 203.84, "text": " The interaction is real-time, and this is really important, and the actions are executed"}, {"start": 203.84, "end": 209.52, "text": " asynchronously while the environment runs at its own time scale. So this means that basically"}, {"start": 210.4, "end": 215.44, "text": " the environment is not a lockstep environment like Atari, where you basically send an action,"}, {"start": 215.44, "end": 221.52, "text": " so the environment waits until you figure out what your next action will be, and then once you send"}, {"start": 221.52, "end": 226.88000000000002, "text": " the action to the environment, it executes a step forward, and then waits again for your next action."}, {"start": 226.88000000000002, "end": 231.68, "text": " So that's the lockstep paradigm. Here on the other hand, it's a real-time environment, that means"}, {"start": 231.68, "end": 237.52, "text": " Android emulator doesn't care whether you skip certain actions if you were too slow to compute"}, {"start": 237.52, "end": 242.88, "text": " that action or whatever was the problem. So that's something that makes this even more challenging"}, {"start": 242.88, "end": 249.12, "text": " than those other simulators. And we'll dig in a bit more details in a couple of minutes there as"}, {"start": 249.12, "end": 254.24, "text": " well, but I just want to state this one. So the sheer number of apps built for a multitude of"}, {"start": 254.24, "end": 258.4, "text": " important aspects of human life ranging from education and business to communication and"}, {"start": 258.4, "end": 264.88, "text": " entertainment provides virtually unlimited challenges for RL research. So this is going"}, {"start": 264.88, "end": 274.0, "text": " to be big in my opinion. We now have a huge taskpad of applications which are actually"}, {"start": 274.0, "end": 279.52, "text": " meaningful, like comparing that to Atari games or those made-up environments. It will be so much"}, {"start": 279.52, "end": 284.8, "text": " easier to actually, after training these agents, to deploy them into some product, so there is a"}, {"start": 284.8, "end": 292.24, "text": " much lesser gap between this environment and the thing you actually want and between compared to"}, {"start": 292.24, "end": 298.08, "text": " Atari or some other environments. Okay, so let's continue here. So I already mentioned, so Android"}, {"start": 298.08, "end": 305.35999999999996, "text": " M is unable to run in lockstep. And let me jump over all of these and first tackle that problem."}, {"start": 305.35999999999996, "end": 311.12, "text": " Okay, so here are some additional details. So they say here, another important factor to consider in"}, {"start": 311.12, "end": 316.0, "text": " real-time environments is that agents require some deliberation time to generate the next action"}, {"start": 316.0, "end": 320.0, "text": " given an observation. So that means if you have an agent that's a convolutional neural network,"}, {"start": 320.0, "end": 324.79999999999995, "text": " once it receives the observation, it will take some time to process that observation in order"}, {"start": 324.8, "end": 330.48, "text": " to output the action. And that doesn't matter in those lockstep environments, but here the timings"}, {"start": 330.48, "end": 335.36, "text": " actually do matter. So they say it here, in a traditional lockstep environment, the environment"}, {"start": 335.36, "end": 341.28000000000003, "text": " generates an observation and pauses to wait until the agent responds with an action before stepping"}, {"start": 341.28000000000003, "end": 350.0, "text": " the simulation forward. So, okay, and the diagram is here and basically, so once you get the"}, {"start": 350.0, "end": 355.2, "text": " observation, they kind of compress the deliberation time because it doesn't matter. So basically,"}, {"start": 355.2, "end": 360.64, "text": " once you compute it, you issue an action and then the environment steps forward and then just repeat."}, {"start": 360.64, "end": 366.56, "text": " And you can see there are no problems here. But on the other hand, if you have a real-time"}, {"start": 366.56, "end": 374.24, "text": " environment like Atari, like Android M, the large deliberation time can be harmful to performance."}, {"start": 374.24, "end": 381.6, "text": " And they have a nice chart here as well. So here the action sends the, so the agent sends the action"}, {"start": 381.6, "end": 387.12, "text": " and you can basically, in order to cope with this problem of this being a real-time environment,"}, {"start": 387.12, "end": 394.8, "text": " you can insert these artificial weights inside your routine. And so what happens is once you"}, {"start": 395.6, "end": 401.36, "text": " issue an action, the simulator starts rendering the observation. So this may vary depending on"}, {"start": 401.36, "end": 406.96000000000004, "text": " the app, et cetera. And then once you get the observation, it takes some time for the agent to"}, {"start": 407.52000000000004, "end": 412.08000000000004, "text": " compute the next action and then it issues the action. And as you can see, because of the"}, {"start": 412.08000000000004, "end": 419.6, "text": " variance of this thing and of the rendering, you'll have to kind of think about this a bit more than"}, {"start": 419.6, "end": 423.44, "text": " in your lockstep environments. So that's something to keep in mind. It's going to make"}, {"start": 424.72, "end": 428.16, "text": " this a bit more challenging, but yeah, I guess it's not that big of a deal."}, {"start": 428.16, "end": 434.96000000000004, "text": " Okay. Let me get back to the beginning of the stack report and let's start with the action"}, {"start": 434.96000000000004, "end": 440.88000000000005, "text": " interface. So basically the action interface is super simple. So the native action space of the"}, {"start": 440.88000000000005, "end": 446.0, "text": " environment consists of a tuple consisting of a position. Basically you have X and Y coordinates"}, {"start": 446.0, "end": 451.92, "text": " which are in the continuous space. And then the Android M will, depending on the actual device"}, {"start": 451.92, "end": 456.96000000000004, "text": " you're using, the resolution of your emulator, it will just discretize these continuous actions."}, {"start": 456.96, "end": 462.0, "text": " Continuous actions into those pixels. So that is, it's basically, you don't have to care about the"}, {"start": 462.0, "end": 468.23999999999995, "text": " actual resolution. The second part of your action will be this discrete value. So it can be touch,"}, {"start": 468.23999999999995, "end": 472.32, "text": " lift, or just repeat. Basically, yeah, you can either touch the screen, you can either lift"}, {"start": 473.59999999999997, "end": 480.47999999999996, "text": " the pointer and you can repeat the previous action. So that's the raw action space. And the thing is"}, {"start": 481.59999999999997, "end": 486.64, "text": " in order to actually, in order to meaningful interact with Android applications, as you may"}, {"start": 486.64, "end": 491.91999999999996, "text": " know, you need these gestures. So which are much more complex. And they say it here, the complexity"}, {"start": 491.91999999999996, "end": 496.71999999999997, "text": " of the interface arises from the fact that individual raw actions on their own do not"}, {"start": 496.71999999999997, "end": 502.4, "text": " necessarily trigger a meaningful change in the environment. And they have nice visualizations"}, {"start": 502.4, "end": 509.36, "text": " here. Basically you can see the swiping and you can see simple touch and lift. You can see a bit"}, {"start": 509.36, "end": 518.08, "text": " more like nonlinear swipe movement. And basically, this makes it so much harder for the agent to learn"}, {"start": 518.08, "end": 524.4, "text": " because you have to have, you have to be both precise in the spatial sense and also in the time"}, {"start": 524.4, "end": 532.08, "text": " sense. So you can't do it like this and then stop here and then continue doing the swipe because the"}, {"start": 532.08, "end": 537.2, "text": " system, the Android itself, probably won't interpret that as a correct gesture as a swipe action. So"}, {"start": 537.2, "end": 542.8000000000001, "text": " you'll have, so the agent will have to both be precise both time-wise and space-wise. Let's call"}, {"start": 542.8000000000001, "end": 548.5600000000001, "text": " it that way. And I think they mention it here. So this need to compose the actions pair with the"}, {"start": 548.5600000000001, "end": 554.0, "text": " difficulty of solving the underlying task itself leads to difficult exploration problem. So for"}, {"start": 554.0, "end": 559.12, "text": " example, in order to learn to play chess, an agent must not only find a winning strategy, it also has"}, {"start": 559.12, "end": 564.5600000000001, "text": " to learn to move pieces through drag and drop gestures. So yeah, that's the part I mentioned."}, {"start": 564.56, "end": 570.9599999999999, "text": " And that's something to keep in mind. Aside from these information from the observation,"}, {"start": 570.9599999999999, "end": 577.52, "text": " which is basically an RGB frame, you'll have additional text task extras. And so an extra"}, {"start": 577.52, "end": 581.1199999999999, "text": " in Android Envy is any information that the environment sends to aid the understanding of"}, {"start": 581.1199999999999, "end": 585.04, "text": " the task. The information sent through this channel is typically very useful for learning,"}, {"start": 585.04, "end": 590.4, "text": " yet difficult to extract from raw pixels. And I gave an example of, I think, like text being"}, {"start": 590.4, "end": 595.28, "text": " displayed on the screen. And instead of you having to parse and learn the OCR additionally,"}, {"start": 596.0799999999999, "end": 602.9599999999999, "text": " you'll have the option of just circumventing that and printing that, like just putting that"}, {"start": 602.9599999999999, "end": 607.92, "text": " into this extras variable, which will contain the string directly, making it a bit easier to learn"}, {"start": 607.92, "end": 617.92, "text": " the task at hand. So I mentioned that in order to actually deploy your RL agent to these applications,"}, {"start": 617.92, "end": 626.0799999999999, "text": " you'll have to modify the apps in a way in order to create the notion of reward and of episode"}, {"start": 626.0799999999999, "end": 632.7199999999999, "text": " signal, et cetera. So they say here, while Android is an operating system with no inherent rewards"}, {"start": 632.7199999999999, "end": 638.3199999999999, "text": " or episodes, Android Env provides a simple mechanism for defining tasks on it. And we'll"}, {"start": 638.3199999999999, "end": 645.1999999999999, "text": " see how that functions. I'll just walk you through the GitHub and that's going to be the best"}, {"start": 645.2, "end": 650.6400000000001, "text": " explanation, I guess. They made a small selection of tasks already, and you can see some apps like"}, {"start": 652.1600000000001, "end": 659.84, "text": " Catch here and RocketSlave, PressButton, some apps like 2048, like this is a popular game,"}, {"start": 659.84, "end": 665.2800000000001, "text": " and Blockinger. And you can see the layout here is pretty complex. And they say it here,"}, {"start": 665.9200000000001, "end": 670.5600000000001, "text": " for example, most agents perform well on tasks such as Catch. So this is a simple one."}, {"start": 670.5600000000001, "end": 674.5600000000001, "text": " They have a simple action interface and dense rewards, whereas the combination of a highly"}, {"start": 674.56, "end": 679.3599999999999, "text": " structured interface, you can see the complex layout here, like all these buttons and here,"}, {"start": 679.3599999999999, "end": 686.4799999999999, "text": " the actual screen of the game. Time sensitivity and sparse rewards renders Blockinger particularly"}, {"start": 686.4799999999999, "end": 692.64, "text": " difficult to solve. And they tested a bunch of common RL agents. And the interesting thing was"}, {"start": 692.64, "end": 697.5999999999999, "text": " that actually DQN, one of the oldest agents out there, like deep learning agents at least,"}, {"start": 698.2399999999999, "end": 703.8399999999999, "text": " deep RL agents, has maybe one of the best performances on all of these tasks. And as you"}, {"start": 703.84, "end": 712.1600000000001, "text": " can see, these cyan curves, DQN outperforms many of these other agents like RTD2 or Impala, etc."}, {"start": 712.1600000000001, "end": 717.0400000000001, "text": " And you can also see the progression, like going from top left here to bottom right, you can see"}, {"start": 717.0400000000001, "end": 724.1600000000001, "text": " that the tasks get harder and harder, and basically the performance really drops down. And yeah."}, {"start": 725.84, "end": 729.36, "text": " Okay, by the way, for those of you who haven't been developing Android apps,"}, {"start": 729.36, "end": 736.88, "text": " let me just kind of give you some intuition here. Basically, when you open up an Android studio,"}, {"start": 736.88, "end": 744.96, "text": " and you can basically in order to debug your app, you initialize this emulator, which can be"}, {"start": 744.96, "end": 753.04, "text": " arbitrary device, which you need to specify, and it will have certain height and certain width of"}, {"start": 753.04, "end": 760.16, "text": " the screen. So you can also pick a specific version of Android. So all of that can be specified"}, {"start": 760.16, "end": 764.56, "text": " through Android Studio. So these are just some things you won't have to worry about because the"}, {"start": 765.36, "end": 773.92, "text": " emulators themselves care about this part. And that's just abstracted from you. Additionally,"}, {"start": 773.92, "end": 783.28, "text": " they mentioned a bunch of useful environments that were previously open sourced or advised."}, {"start": 783.28, "end": 789.8399999999999, "text": " Some of them are like you have to pay the license like Mojoco. But they end up here saying that"}, {"start": 789.8399999999999, "end": 794.24, "text": " since Android has billions of users and Android M provides tasks that run on the standard Android"}, {"start": 794.9599999999999, "end": 800.0, "text": " operating system simulator, agents trained on the platform could potentially tackle a wide range of"}, {"start": 800.0, "end": 805.28, "text": " use cases leading to direct real world impact. And this is something I'm really passionate about. So"}, {"start": 805.28, "end": 813.12, "text": " seeing these agents and RL in general, having some real world impact will be super, super interesting."}, {"start": 813.12, "end": 820.64, "text": " And I hope to see like research and apps happening because of this new environment. So having said"}, {"start": 820.64, "end": 826.08, "text": " that, let me go back to the GitHub and show you how to actually create a task. Okay, so if you open"}, {"start": 826.08, "end": 834.32, "text": " up this tasks guide MD file, you'll see these two sections here where they basically describe what"}, {"start": 834.32, "end": 840.4000000000001, "text": " you need to do in order to enable your Android application to be used as a training ground for"}, {"start": 840.4000000000001, "end": 848.64, "text": " your your RL agent. And the main things are so you have this setup steps, which will once the once"}, {"start": 848.64, "end": 853.5200000000001, "text": " the training start, it will just do a couple of things like you can automate installation of the"}, {"start": 853.52, "end": 860.96, "text": " the application from the APK APK file, which is just Android binary. You can do a bunch of those"}, {"start": 860.96, "end": 868.3199999999999, "text": " stuff. And I'll just focus on the stuff important to RL. And that's, that's defining rewards and"}, {"start": 869.36, "end": 874.8, "text": " like end of episode signal. So you'll basically have to have to set up this log parsing config"}, {"start": 875.6, "end": 882.3199999999999, "text": " in order to just define certain regs expressions. Because the way that apps will communicate"}, {"start": 882.32, "end": 888.6400000000001, "text": " the reward information, as I think I mentioned, is through the log log, basically through the log"}, {"start": 888.6400000000001, "end": 896.08, "text": " console. So here's an example of a text proto file, which defines the task. And if I expand it,"}, {"start": 896.08, "end": 900.8000000000001, "text": " you can see so it's pretty self explanatory. So perform these upon launching the environment,"}, {"start": 900.8000000000001, "end": 905.44, "text": " install the app, check if it's installed correctly, etc. So that's the not that interesting part. Let"}, {"start": 905.44, "end": 911.44, "text": " me just skip down to the this part. And basically, you can see you define the regx expression for"}, {"start": 911.44, "end": 920.6400000000001, "text": " your reward and for your score for the episode and signal etc. And the way you know this all works is"}, {"start": 920.6400000000001, "end": 925.5200000000001, "text": " the following. So you might you might have noticed the tasks often rely on log messages exposed by"}, {"start": 925.5200000000001, "end": 930.72, "text": " the Android system, which Android app can intercept and translate into items such as rewards,"}, {"start": 930.72, "end": 936.24, "text": " episode and signals or task extras. And they said here, so of course, applications might not send"}, {"start": 936.24, "end": 942.0, "text": " suitable messages by default. So in order to have access to such messages, we often add them to the"}, {"start": 942.0, "end": 947.76, "text": " app source code to match our expectation. For example, in the case of the 2048 app, we find in"}, {"start": 947.76, "end": 952.64, "text": " the game source code the exact lines where the score is computed, and add a line to log this value"}, {"start": 952.64, "end": 959.44, "text": " in the format that is expected by the text proto. So that's this file here. So as you can see here,"}, {"start": 959.44, "end": 963.6800000000001, "text": " some regular expressions, we have a plus or minus sign, then we have a couple of digits"}, {"start": 963.68, "end": 968.88, "text": " followed by a dot and then a couple of digits more. So basically a float like a scalar value."}, {"start": 969.76, "end": 974.4, "text": " And so you add the line to log this value in the format that is expected by the text proto,"}, {"start": 974.4, "end": 978.9599999999999, "text": " or conversely, make sure that yeah, whatever. So basically, you'll have to edit the source"}, {"start": 978.9599999999999, "end": 985.1999999999999, "text": " code of your application by adding a logging line here. And then through this text proto,"}, {"start": 985.1999999999999, "end": 989.68, "text": " you'll be able to send that environment to the agent. And that's how you're going to actually"}, {"start": 989.68, "end": 994.2399999999999, "text": " train and set up everything. And finally, they have a couple of examples here on this page,"}, {"start": 994.9599999999999, "end": 1000.4799999999999, "text": " example tasks. And I'll just link all of these in the description. But basically, here is a really"}, {"start": 1000.4799999999999, "end": 1006.8, "text": " simple environment they created. And in this one, so take like, take this one as an example. So"}, {"start": 1006.8, "end": 1012.56, "text": " basically, the RL agent gets a reward plus one, if it clicks the B button, and if it clicks the A"}, {"start": 1012.56, "end": 1019.12, "text": " button, it just the episode ends, because that's how they defined it. And basically, what happens"}, {"start": 1019.12, "end": 1027.36, "text": " then is the app shuts down, and the environment will automatically launch relaunch the app again."}, {"start": 1027.36, "end": 1033.84, "text": " And that's done through all through the task proto file we just saw. So that's how it works. And"}, {"start": 1035.44, "end": 1039.76, "text": " there are, aside from this one, they have a bunch of other examples, like with a calculator,"}, {"start": 1040.32, "end": 1048.72, "text": " some games, like the 2084 dimension, for which they had to modify the source code, and like chess."}, {"start": 1048.72, "end": 1053.6000000000001, "text": " So yeah, bunch of different stuff. And yeah, hopefully you found this one useful. If you did,"}, {"start": 1053.6, "end": 1081.6, "text": " leave a like, subscribe and see you in the next video."}]