Implementing DeepMind's DQN from scratch! | Project Update
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

I walk you through my project (it's still not working - RL of course it's not working!) and explain to you my workflow, how I think, organize my project, todos, etc.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ My GitHub: https://github.com/gordicaleksa
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro, my other deep learning projects
01:55 Patreon
02:15 Tensorboard walk-through
05:08 Code walk-through - understand DQN arguments
07:25 Main loop (collecting experience and learning from it)
14:15 Main actor-learner class
20:19 Visualizations (matplolib, tensorboard)
23:30 Analyzing other people's projects

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#dqn #projectupdate #reinforcementlearning
 What's up? In this video I'm going to give you an update of my project. So for those of you who don't know, I've been developing the DeepQ network project from DeepMind from scratch. And I thought of showing you how the code looks like before I actually open source it, when it's in a completely polished form. So it's still not done. Like basically I did wrote most of the code already, but it's not working because it's RL. And I'm still trying to debug it. So I want to walk you through how I structure the project, how I test individual components, which to do. So in general, what my workflow is, and I still have a bunch of to-dos in the code, as you'll see. So I also show you the the the logs I've been doing. So I've been logging like stuff like even grad gradient norms because I'm experiencing vanishing gradients problems. So yeah, hopefully you'll get something out of it. If you do, please share the video and subscribe. Before I jump into the code, for those of you who are new to this channel, I want to show you this. And basically my GitHub profile. And I want to just mention that I've got a lot of projects that I've been working on and open sourcing over the last year and three or four months. And so do check them out. So by far the most popular one is the PyTorch, like the GraphAttention network implementation. So do check that out if you are interested in GraphML. And I also have like a neural style transfer. I have like three repos there for like the slow one, the fast one, and even one for videos. Then I've got this Deep Dream repo. And this pick you can see here is basically was created using the software I developed for Deep Dream. I also have this GAN project. So check it out as well. And finally, VASFANI Transformers, so the original transformer paper from 2017. So yeah, and as you can see here, looking at my log, I've been working on deep key networks for almost two weeks now. And as I said, it's still not working. And as a final note, I do have a Patreon account. So if you want to support me financially, even though it's completely optional, if you're already sharing my content and you like my content, that's already super enough. But having said that, if you still want to support me like financially, you can do that on my Patreon. I'll link it in the description. So let's jump into the actual code. Okay. So first things first, I'll assume you already know what DQN is. And if you haven't watched the video, I actually covered DQN. I'll link it somewhere here in one of my previous videos. So do check that out. And now the thing is, because I have TensorBoard, I'll just kick off and start the run so that we can actually see the logs I'm collecting while I'm explaining the code. So let me just briefly walk you through what I'm logging. The first thing is like epsilon, and that's the hyperparameter in the epsilon greedy algorithm, which DQN uses. Then we have FPS. I'm basically logging the number of frames. So the number of steps, environment steps I do per second. And it's usually around 400, which is pretty nice. Then I'm logging some metrics. So these just tell me how fast my training will end up. Epsilon is not that actually useful, because it's just that it's correct. I just want to see that it's correct pretty much. So HuberLoss is the loss that's used for DQN. So it's basically a mix of mean square error loss and L1 when you go outside the minus one, one range, simple stuff. Then we have rewards per episode. And I'm not doing any averaging here. So I'm just logging the cumulative reward I get per episode. And as you can see, it's just floating around minus 21, which means the algorithm is not learning anything, although I do have a little only a little bit of samples here. Anyways, steps per episode just tells me how much steps do I need before the episode terminates. And finally, I'm actually logging these gradients, the norms of gradients, because I'm experiencing experiencing. So I saw that the network is not learning for some reason. So I thought just plotting the gradients and I saw I'm having a vanishing gradients problem. So if you take a look at the DQN network, let me just find some picture of DQN. So this one, so you can see there are three convolutional, so you can only see two here, but there are three convolutional neural layers in DQN. And there are two fully connected layers. So that's five in total. And if you take a look at the plots, that's exactly what you see. So you have the first convolutional layer, the second, the third, then we have the fully connected and then have the total gradient here. So I basically just plot the weights and the biases per layer. So that's why we have 10. And then the final one, I just get by combining, by concatenating all of the parameters together and finding the L2 norm. And yeah, you can see that it's not learning anything because the gradients are almost zero. And let me just toggle on the newest run. So that's the one that's currently running. And we'll later see the trends, but like, yeah, for the time being, just let it run. And let's focus on the code. So first things first, yeah, I just have some arguments for the program, which I then wrap into this dictionary and I pass it into the main function. Let me just walk you through, briefly walk you through this, like what the parameters are. So first things first is the seed thing. And if you're not doing, if you've never done RL, you probably, you've seen seed occasionally, but it's not that vital. Here on the other hand, it's very super important that you include the seed because you want to have some sort of ability to reconstruct, to reproduce the results you got. Because if you're familiar with RL, basically it's very noisy. So you can have maybe in 60% of the seeds, you get some nice results. And then maybe 40%, you get like zero. And that sucks. So once you find a seed that works, you want to have the number. And yeah, so that's the idea there. I'm using punk environment and the no frame skip part means basically because I'm using some wrappers from this stable baselines three library. I don't want to skip any frames because the wrapper itself will be skipping frames. And if you're not familiar why we're skipping frames, that's because in Atari, you basically, the DeepMind crew back in 2013, just made this convention that it's better to, it's a trade off between performance and results you'll get. And basically you just act on every fourth frame, and then you just repeat the actions four times. So you act on the first frame, then you just repeat the actions and then you take the fourth frame and then yeah, so you're just skipping the intermediate frames and you're doing this dummy action by just repeating the action. So that's that part before just means that the environment is deterministic. So if you take we zero, I think basically then the environment will sometimes execute your action, but sometimes 25% probability, you will execute one of the previous sections, actually the previous section. Yeah, a number of steps, 50 million, that's the DQN hyperparameter. These are not that important. Replay buffer, yeah, nothing super important here. And I just have some logging parameters. So let me just walk you through the actual main function. And that's important. Okay, so first thing I do is you can see here, I take the NYD, so that's the Pong no frame, V4 string, and I just wrap it in this wrapper. And that's a common workflow you'll see across many projects. So that's the only part of the code that I have reused from a different library and that's stable baselines three in this case. And I additionally just had to create this one more transform, which would just swap the access and because the DQN in PyTorch basically requires that the channel comes first. So what I do is I just take the height, width, and channel and just permute the dimensions so that I get the channel first tensor. And that's pretty much it. Monitor just basically tracks your different statistics from your training, like the rewards you get per episode, like the length of the episode, you can even record videos, but I'm not doing that at the moment. So that's it. Okay, then once we have the environment, so that's the OpenAI Gym environment, I create a buffer. And basically the reason I had to develop buffer from scratch is because stable baselines three, and this may not sound that intuitive because it's a really nice library, they don't support a smart buffer. And what I mean by smart buffer is that they actually have the when you want to store 1 million frames, what they actually do is instead of storing 1 million and then so basically, instead of storing, let me write it here. So 1m, and then maybe 184, 84 are the usual like height and width dimensions, they're actually storing 1m, then 1, 4, 84, 84. And the reason is observations for DQN are just basically you stack the last four frames. And that's what you pass through the network because the states are what they call they call that perceptually aliased. That means you do need previous frames in order to understand the state you're in and to make this a Markov decision process. Okay, so because they don't have the smart buffer, I implemented the replay buffer from scratch. And yeah, secondly, set random seeds is really important. And I mentioned that already. But let me just show you here. So we both we have to set the seed for both the torch, the pytorch framework for the NumPy library for Python's inner random library, then for OpenAI.GM and both for the action space as for the GMes itself, because there was some weirdo bug. And for some reason, you'd expect this to set the action space as well, but it did not. And so I have to do that as well. So yeah, just keep that in mind, setting seeds in RL agent is very important. So you can actually reproduce results and have some stable trends. Linear schedule, simple class, which helps me anneal the epsilon hyperparameter from one to 01 over the span of certain number of steps. And that's just the thing that the paper used. So if I go to linear schedule, you can see it here. And the reason I'm actually going here is because I want to show you what I usually do when I develop a project. So you see the class here. And in order to be sure that at least some components are working, I create these subtests in the file itself. So you can see here, I actually instantiate those linear schedule and I test it. So if I run the utils, you'll see that, yeah, we have correct results. So it goes from one to 01 over the span of 50 steps. And the reason it's 50 is because I've set 50 here. So that's the way I test components. So first, most of the components are usually orthogonal, you can test them. And then once you combine them in the complete agent, infrastructure, you want to be confident in some components so that you can debug. Yeah. And even if, as you can see, even though I've done this, I'm still having problems debugging the RL agent. So testing is very important and just be pedantic when you're developing RL code. Okay. Basic stuff here, PyTorch. If I have GPU and I have RTX 2080, I just fetch here the device. I instantiate the DQN network, the target DQN, which we use to set the TDD targets in the DQN algorithm. And finally, this is the main thing, this ActorLearner class is what I used, is a nice abstraction and you'll see how it looks like. But basically you can see the shortest training loop ever here. And it's pretty intuitive. So basically until you have, so get experience count, so until you have collected this number of experiences, so frames, we just keep on collecting experience and we keep on learning. So first we go and collect experience. So we'll have an inner loop of four steps where we are collecting, interacting with the environment, taking the rewards, the down flags, the frames, observations, and we'll be storing them in a replay buffer. And once we have enough data, and there is this delay here, we start learning. So basically how DQN works is you initially don't learn, you're just collecting data and filling the replay buffer. And once you get to this point, now I start learning from the experience. And here I'm just sampling from the buffer and doing the HuberLoss stuff. Okay, let me continue. So along the way, I do note some stuff that's weird, that are weird, and some stuff that I want to do. So go through the code once more and make sure there are no bugs, it's an obvious one. But like this thing is something that I'm still confused about. For some reason, the game of Pong, even though you basically only need three actions, that's move up, move down, or no up, or basically just stay in place. For some reason, this Atari learning environment, the ALE, is giving me six outputs. If somebody knows it already, just write it down in the comments. But yeah, so that's how I think. So once I see something that's problematic, I just write it down in the code itself, using these to-dos. And this ID, the byte arm, basically highlights them in this yellow color. So it's really easy for me to organize them. And I also have the to-do list here, so I can just organize my to-dos really nicely. So that's cool. Having said that, let's go to this ActualLearner class. Let me show you, and again, so a bunch of to-dos here, as you can see. And so ActualLearner, the init function is just basically a wrapper class. So I just store all of these replay buffers, dqns, etc., as like these private variables in Python. And what I do here is I just log the initial time of the program, because I'll be using this to calculate the frame per second metric. And yeah, so basically, just a container, the init function just takes care of these standard stuff. I just instantiate the loss here. I create the atom optimizer. And as you can see here, so the original paper used RMS prop, but that was before Atom was published, so I guess that's why they didn't use Atom. So I just, I'm using Atom, and if I'm having problems because of Atom, I'll try out RMS prop as well. And yeah, a short note to myself just to kind of freshen up my knowledge of how those algorithms exactly work, so that I can better debug this code. Okay, so first things first, collect experience. So that's one of the main functions. And as you can see here, so this number is equals four, so we just collect the experiences four times, so four steps in the environment. And what we do here, we just store the frame in the buffer, then we fetch the last observation, which will concatenate the four last frames. And if it doesn't have enough data, it will just fill in the black frames before. So yeah, basically, at the end, you'll have four frames, and some of them may be black if we don't have enough data. Then we sample the action using that observation. And that function just wraps up a couple of details from the DQN algorithm. It's a neat way to be concise here. And what I do is, initially, before the learning starts, and if you remember this part, this is the same thing as this part here. So we wait for 50k steps, and then we start learning. And before we start learning, we have, we just take random actions from the environment. So that means we'll be acting totally randomly in the environment initially. Then once that, once we have enough steps, we'll start doing something smart. And that's actually exploiting DQN's Epsilon greedy, as you can see here. So with Epsilon probability, so we just roll the dice with Epsilon probability, we randomly act. Otherwise, we act greedily. We take the dead action that corresponds to the Q value that's the highest. And yeah, that's it. Let me get back to the algorithm sample action. So we pick the action, we send the action to the environment, and that's your classical RL feedback loop. And I get the new frame back, the rewards and the dumb flag in case the episode is done or I've lost the life. Then what I do, I store effects. So basically, if you remember, I first stored a frame, but only now do I have the action and the reward and the dumb flags. So at the same index, I just add that data so that we can later fetch it during the training, during the training when we are sampling from the buffer. Simple stuff here. And yeah, I'm logging some stuff here, as you can see. So some of the logging utilities are logging depending on the like every fourth episode. And some logging is just coupled to the number of steps and not to the number of episodes because it makes more sense and you'll see why. Okay, the second important function is learning from experience. And you can see it here, basically, what I do here is I fetch from the replay buffer the batch size, which is 32 in the algorithm. And so we have all of we get the data as you can see observations, actions, rewards, next observations and dumb flags. And we just create the basically the target values, as you can see here, so it packs the next observation into the target DQN, I find the max values, max values, and we just add the rewards and scale with the discount factor gamma, we scale those numbers. And this thing here may be interesting. So basically, when the dumb flag is one, you want to make sure that this will zero out and this will be zero when the dumb flag is one, and we'll just have the reward. Why is that? Well, because basically, you don't want to have the you don't want to pass the final state, the terminal state into DQN, we just have the reward, we don't want to introduce the noise there because the state is already done. Basically, what the Q learning function does, it predicts the expected reward. And since the state is already since the game is done, the expected reward should be zero. And yeah, that's why we zero out that part. Then we have the observations, we pass them through DQN. And we just take the actions that we sampled during the experience collection. And we take those Q values. And finally, we just have the loss between the target values. And so that's your DQN algorithm. And this part is not that important if you didn't understand. I'm just walking you through how I'm thinking and how I'm structuring the code and everything. Okay, then basic stuff. We're zeroing the gradient gradients, we do the backward, which will accumulate the grad fields in PyTorch. So every single variable will have the grad fields filled in with the gradient. And we do the step. So that's it. That's how we learn. Two more details. First off, every fourth update, so no, actually every 10k update, I'll be doing a hard update of the target network. And that's what the original paper showed, improved the stability significantly. So aside from replay buffer, this thing is very important in order to have an algorithm that's stable and that can work. So finally, you can see I'm logging gradients here. And that's super important because I'm still trying to debug this thing. And yeah, okay, so I think I've got enough data in the tensor board. So I'll just go ahead and stop this run. And what I'll do, I want to show you what the visual observation is. Basically, collect experience, I'll just uncomment this thing here. I'll put a breaking point there and I'll rerun this thing. So let me show you what this does. Whoops, that's the wrong file. I'm starting the main file. Okay, if I go here and we just plot the function, you can see initially because we only have one frame, and this is the observation. So the last three frames will be black. Okay, so if we go and do another step, so as you can see, we did another step. I step into here, I plotted, we now have two frames. So if I do another step again, we did another step and I step in, you can see we have three frames. So that's what I have this for. So just a simple function which will help me better understand what's going on. And okay, let me show you the logging functions in tensor board and we're pretty much done. When it comes to logging, I'm just, I'm basically logging the stuff you already saw in the tensor board. So reward per episode, blah, blah, blah, nothing smart there. So maybe the only a bit more interesting part is here. So I'm iterating through the parameters of the decan model and I'm just calculating the, as you can see, I take the gradient, the data, and I did the norm, the L2 norm, P equals two L2 norm, and I just basically log those numbers. And finally, I'm accumulating these numbers in the total grad so we can have the complete vector and not just the parameters for a specific layer, but for the whole network itself. Okay, let me show you the tensor board finally and see what the data we've got. Okay, Epsilon as expected, just linearly annealed to 0.1. That's not so interesting. FPS, as you can see, it's pretty stable, around 330 frames. I don't know whether that's good or bad because I haven't been experimenting enough, but we'll see. That sounds like a nice number. Huber loss, as you can see, it's just random noise pretty much. I don't see any downward trends, so something's wrong, or maybe I just didn't have enough steps. That's also a possibility. Also rewards, as you can see, I'm almost constantly at minus 21. That means I basically, the agent loses all of the 21 games of Pong against the built-in agent, so that means it's just playing really bad. Steps per episode, again, really noisy, and you can see the number of steps that's needed per episode, so that's around maybe the mean is around 850 or something. And finally, we can see the grads, and they are completely zero. So if you've seen the bug in my code, comment down in the comment section. I'll hopefully have this thing running in a couple of days, and then I'll need a couple more days to just polish everything, create a readme, so that people can kind of use it and not experience the problems I did. So yeah, one more thing I want to mention, which is really important, how I actually go about developing these projects. So usually, unless nobody has published any code whatsoever, which I did have a couple of occasions where I had to implement a paper totally from scratch, I just had the paper, so what I ended up doing is pinning the entire authors and communicating with them, so I did get some snippets there. So basically, you always want to start with some code and understand what other people are doing. So I did investigate at least five, six, or seven, something like that, PyTorch implementations, I think six. And one of them here is stable baselines, so I was just stepping through the code, looking what those folks did, and as I said, they didn't have the smart replay buffers, so I had to implement the buffer from scratch, and stuff like that. But I did learn some nice tricks from this repo, so that's a cool thing to do when you're developing a project. Go ahead and first analyze the other projects, and you'll learn some stuff additionally, and it will be a much more rewarding experience. You'll learn more if you also understand what other people have been doing. Okay, having said that, maybe the last thing I want to show you are quickly the DQN model, what I've done there. And you can see I just created a super generic model there, so you can play with it, and you'll have the code in a couple of days, hopefully until next weekend, actually. And you can see I have this automatic part here, where because, as you know, we have three convolutional layers, we have fully connected layer, that means that basically you can't change the input, because otherwise the compatibility between the volume that comes out from the third comb layer won't be compatible with the first fully connected layer, so you either have to keep the observation constant, or you can do it like me here. So I'm basically passing the input volume, and I can automatically calculate the number of neurons that I need in the first fully connected layer. So that's this part here. And yeah, that's not an interesting replay buffer, on the other hand. As you can see, the main thing I've done it is because I only need 7GB of memory, only of RAM, instead of 28 for the implementation that these stable baselines have. And yeah, basically a couple of public functions here, and the bunch of work happens here. There is not a lot of code, but there are certain edge cases, and the implementation I actually took an inspiration from was this Berkeley's implementation, and they also have a small bug. It's a subtle one, it won't actually impact the final result of the Dequeon agent, but nonetheless I did found it and submit an issue. And yeah, that's pretty much it. Hopefully you found this video useful. If you did, leave a comment, subscribe, share the video, and I hope to see you next time.
[{"start": 0.0, "end": 4.72, "text": " What's up? In this video I'm going to give you an update of my project. So for those of you who"}, {"start": 4.72, "end": 10.64, "text": " don't know, I've been developing the DeepQ network project from DeepMind from scratch."}, {"start": 10.64, "end": 15.120000000000001, "text": " And I thought of showing you how the code looks like before I actually open source it,"}, {"start": 15.120000000000001, "end": 19.92, "text": " when it's in a completely polished form. So it's still not done. Like basically I did"}, {"start": 19.92, "end": 25.52, "text": " wrote most of the code already, but it's not working because it's RL. And I'm still trying"}, {"start": 25.52, "end": 29.84, "text": " to debug it. So I want to walk you through how I structure the project, how I test individual"}, {"start": 29.84, "end": 35.76, "text": " components, which to do. So in general, what my workflow is, and I still have a bunch of to-dos"}, {"start": 35.76, "end": 40.72, "text": " in the code, as you'll see. So I also show you the the the logs I've been doing. So I've been"}, {"start": 40.72, "end": 47.6, "text": " logging like stuff like even grad gradient norms because I'm experiencing vanishing gradients"}, {"start": 47.6, "end": 53.2, "text": " problems. So yeah, hopefully you'll get something out of it. If you do, please share the video and"}, {"start": 53.2, "end": 57.28, "text": " subscribe. Before I jump into the code, for those of you who are new to this channel, I want to show"}, {"start": 57.28, "end": 65.2, "text": " you this. And basically my GitHub profile. And I want to just mention that I've got a lot of"}, {"start": 65.2, "end": 68.8, "text": " projects that I've been working on and open sourcing over the last year and three or four"}, {"start": 68.8, "end": 75.12, "text": " months. And so do check them out. So by far the most popular one is the PyTorch, like the"}, {"start": 75.12, "end": 80.4, "text": " GraphAttention network implementation. So do check that out if you are interested in GraphML. And I"}, {"start": 80.4, "end": 85.92, "text": " also have like a neural style transfer. I have like three repos there for like the slow one,"}, {"start": 85.92, "end": 91.12, "text": " the fast one, and even one for videos. Then I've got this Deep Dream repo. And this pick you can"}, {"start": 91.12, "end": 97.36000000000001, "text": " see here is basically was created using the software I developed for Deep Dream. I also have"}, {"start": 97.36000000000001, "end": 103.04, "text": " this GAN project. So check it out as well. And finally, VASFANI Transformers, so the original"}, {"start": 103.04, "end": 109.68, "text": " transformer paper from 2017. So yeah, and as you can see here, looking at my log, I've been"}, {"start": 109.68, "end": 115.84, "text": " working on deep key networks for almost two weeks now. And as I said, it's still not working. And"}, {"start": 115.84, "end": 121.12, "text": " as a final note, I do have a Patreon account. So if you want to support me financially, even though"}, {"start": 121.12, "end": 127.60000000000001, "text": " it's completely optional, if you're already sharing my content and you like my content,"}, {"start": 127.60000000000001, "end": 132.72, "text": " that's already super enough. But having said that, if you still want to support me like financially,"}, {"start": 132.72, "end": 137.44, "text": " you can do that on my Patreon. I'll link it in the description. So let's jump into the actual code."}, {"start": 137.44, "end": 144.07999999999998, "text": " Okay. So first things first, I'll assume you already know what DQN is. And if you haven't"}, {"start": 144.07999999999998, "end": 149.68, "text": " watched the video, I actually covered DQN. I'll link it somewhere here in one of my previous videos."}, {"start": 149.68, "end": 155.92, "text": " So do check that out. And now the thing is, because I have TensorBoard, I'll just kick off"}, {"start": 155.92, "end": 166.32, "text": " and start the run so that we can actually see the logs I'm collecting while I'm explaining the code."}, {"start": 166.32, "end": 170.79999999999998, "text": " So let me just briefly walk you through what I'm logging. The first thing is like epsilon,"}, {"start": 170.79999999999998, "end": 176.88, "text": " and that's the hyperparameter in the epsilon greedy algorithm, which DQN uses. Then we have"}, {"start": 176.88, "end": 182.79999999999998, "text": " FPS. I'm basically logging the number of frames. So the number of steps, environment steps I do per"}, {"start": 182.79999999999998, "end": 188.56, "text": " second. And it's usually around 400, which is pretty nice. Then I'm logging some metrics."}, {"start": 189.76, "end": 195.44, "text": " So these just tell me how fast my training will end up. Epsilon is not that actually useful,"}, {"start": 195.44, "end": 200.88, "text": " because it's just that it's correct. I just want to see that it's correct pretty much."}, {"start": 200.88, "end": 206.88, "text": " So HuberLoss is the loss that's used for DQN. So it's basically a mix of mean square error loss"}, {"start": 206.88, "end": 212.16, "text": " and L1 when you go outside the minus one, one range, simple stuff. Then we have rewards per"}, {"start": 212.16, "end": 218.72, "text": " episode. And I'm not doing any averaging here. So I'm just logging the cumulative reward I get"}, {"start": 218.72, "end": 225.6, "text": " per episode. And as you can see, it's just floating around minus 21, which means the algorithm is not"}, {"start": 225.6, "end": 230.24, "text": " learning anything, although I do have a little only a little bit of samples here. Anyways,"}, {"start": 231.2, "end": 238.16, "text": " steps per episode just tells me how much steps do I need before the episode terminates. And finally,"}, {"start": 238.16, "end": 245.36, "text": " I'm actually logging these gradients, the norms of gradients, because I'm experiencing experiencing."}, {"start": 245.36, "end": 250.56, "text": " So I saw that the network is not learning for some reason. So I thought just plotting the gradients"}, {"start": 250.56, "end": 255.12, "text": " and I saw I'm having a vanishing gradients problem. So if you take a look at the DQN network,"}, {"start": 255.12, "end": 263.12, "text": " let me just find some picture of DQN. So this one, so you can see there are three convolutional,"}, {"start": 263.12, "end": 268.24, "text": " so you can only see two here, but there are three convolutional neural layers in DQN."}, {"start": 268.24, "end": 272.08000000000004, "text": " And there are two fully connected layers. So that's five in total. And if you take a look"}, {"start": 272.08, "end": 277.12, "text": " at the plots, that's exactly what you see. So you have the first convolutional layer, the second,"}, {"start": 277.12, "end": 282.15999999999997, "text": " the third, then we have the fully connected and then have the total gradient here. So I basically"}, {"start": 282.15999999999997, "end": 286.96, "text": " just plot the weights and the biases per layer. So that's why we have 10. And then the final one,"}, {"start": 286.96, "end": 293.36, "text": " I just get by combining, by concatenating all of the parameters together and finding the L2 norm."}, {"start": 293.36, "end": 300.15999999999997, "text": " And yeah, you can see that it's not learning anything because the gradients are almost zero."}, {"start": 300.16, "end": 304.0, "text": " And let me just toggle on the newest run. So that's the one that's currently running."}, {"start": 304.0, "end": 308.72, "text": " And we'll later see the trends, but like, yeah, for the time being, just let it run."}, {"start": 309.6, "end": 317.44000000000005, "text": " And let's focus on the code. So first things first, yeah, I just have some arguments for the"}, {"start": 317.44000000000005, "end": 321.92, "text": " program, which I then wrap into this dictionary and I pass it into the main function. Let me"}, {"start": 321.92, "end": 326.0, "text": " just walk you through, briefly walk you through this, like what the parameters are. So first things"}, {"start": 326.0, "end": 332.48, "text": " first is the seed thing. And if you're not doing, if you've never done RL, you probably, you've seen"}, {"start": 332.48, "end": 337.28, "text": " seed occasionally, but it's not that vital. Here on the other hand, it's very super important that you"}, {"start": 337.84, "end": 345.76, "text": " include the seed because you want to have some sort of ability to reconstruct, to reproduce the"}, {"start": 345.76, "end": 353.36, "text": " results you got. Because if you're familiar with RL, basically it's very noisy. So you can have"}, {"start": 353.36, "end": 359.6, "text": " maybe in 60% of the seeds, you get some nice results. And then maybe 40%, you get like zero."}, {"start": 359.6, "end": 365.12, "text": " And that sucks. So once you find a seed that works, you want to have the number. And yeah,"}, {"start": 365.12, "end": 371.04, "text": " so that's the idea there. I'm using punk environment and the no frame skip part means"}, {"start": 372.16, "end": 376.64, "text": " basically because I'm using some wrappers from this stable baselines three library."}, {"start": 376.64, "end": 380.72, "text": " I don't want to skip any frames because the wrapper itself will be skipping frames."}, {"start": 380.72, "end": 385.44000000000005, "text": " And if you're not familiar why we're skipping frames, that's because in Atari, you basically,"}, {"start": 385.44000000000005, "end": 393.12, "text": " the DeepMind crew back in 2013, just made this convention that it's better to, it's a trade off"}, {"start": 393.12, "end": 401.20000000000005, "text": " between performance and results you'll get. And basically you just act on every fourth frame,"}, {"start": 401.20000000000005, "end": 406.8, "text": " and then you just repeat the actions four times. So you act on the first frame, then you just repeat"}, {"start": 406.8, "end": 411.28000000000003, "text": " the actions and then you take the fourth frame and then yeah, so you're just skipping the"}, {"start": 411.28000000000003, "end": 415.36, "text": " intermediate frames and you're doing this dummy action by just repeating the action."}, {"start": 415.36, "end": 421.2, "text": " So that's that part before just means that the environment is deterministic. So if you take we"}, {"start": 421.2, "end": 427.44, "text": " zero, I think basically then the environment will sometimes execute your action, but sometimes 25%"}, {"start": 427.44, "end": 432.40000000000003, "text": " probability, you will execute one of the previous sections, actually the previous section. Yeah,"}, {"start": 432.4, "end": 438.32, "text": " a number of steps, 50 million, that's the DQN hyperparameter. These are not that important."}, {"start": 439.2, "end": 444.79999999999995, "text": " Replay buffer, yeah, nothing super important here. And I just have some logging parameters."}, {"start": 444.79999999999995, "end": 449.84, "text": " So let me just walk you through the actual main function. And that's important. Okay, so"}, {"start": 450.96, "end": 456.08, "text": " first thing I do is you can see here, I take the NYD, so that's the Pong no frame,"}, {"start": 456.08, "end": 463.44, "text": " V4 string, and I just wrap it in this wrapper. And that's a common workflow you'll see"}, {"start": 464.0, "end": 470.15999999999997, "text": " across many projects. So that's the only part of the code that I have reused from a different"}, {"start": 470.15999999999997, "end": 476.79999999999995, "text": " library and that's stable baselines three in this case. And I additionally just had to create this"}, {"start": 476.79999999999995, "end": 482.88, "text": " one more transform, which would just swap the access and because the DQN in PyTorch basically"}, {"start": 482.88, "end": 488.48, "text": " requires that the channel comes first. So what I do is I just take the height, width, and channel"}, {"start": 488.48, "end": 494.96, "text": " and just permute the dimensions so that I get the channel first tensor. And that's pretty much it."}, {"start": 494.96, "end": 501.44, "text": " Monitor just basically tracks your different statistics from your training, like the rewards"}, {"start": 501.44, "end": 506.88, "text": " you get per episode, like the length of the episode, you can even record videos, but I'm"}, {"start": 506.88, "end": 513.28, "text": " not doing that at the moment. So that's it. Okay, then once we have the environment, so that's the"}, {"start": 513.28, "end": 521.76, "text": " OpenAI Gym environment, I create a buffer. And basically the reason I had to develop buffer from"}, {"start": 521.76, "end": 527.12, "text": " scratch is because stable baselines three, and this may not sound that intuitive because it's a"}, {"start": 527.12, "end": 532.16, "text": " really nice library, they don't support a smart buffer. And what I mean by smart buffer is that"}, {"start": 532.16, "end": 539.6, "text": " they actually have the when you want to store 1 million frames, what they actually do is instead"}, {"start": 539.6, "end": 546.48, "text": " of storing 1 million and then so basically, instead of storing, let me write it here. So 1m,"}, {"start": 547.12, "end": 555.52, "text": " and then maybe 184, 84 are the usual like height and width dimensions, they're actually storing 1m,"}, {"start": 555.52, "end": 565.52, "text": " then 1, 4, 84, 84. And the reason is observations for DQN are just basically you stack the last four"}, {"start": 565.52, "end": 571.52, "text": " frames. And that's what you pass through the network because the states are what they call"}, {"start": 571.52, "end": 577.52, "text": " they call that perceptually aliased. That means you do need previous frames in order to understand"}, {"start": 577.52, "end": 583.92, "text": " the state you're in and to make this a Markov decision process. Okay, so because they don't"}, {"start": 583.92, "end": 588.56, "text": " have the smart buffer, I implemented the replay buffer from scratch. And yeah,"}, {"start": 590.24, "end": 596.3199999999999, "text": " secondly, set random seeds is really important. And I mentioned that already. But let me just show"}, {"start": 596.3199999999999, "end": 602.8, "text": " you here. So we both we have to set the seed for both the torch, the pytorch framework for the"}, {"start": 602.8, "end": 610.56, "text": " NumPy library for Python's inner random library, then for OpenAI.GM and both for the action space"}, {"start": 610.56, "end": 615.28, "text": " as for the GMes itself, because there was some weirdo bug. And for some reason, you'd expect"}, {"start": 615.28, "end": 620.2399999999999, "text": " this to set the action space as well, but it did not. And so I have to do that as well. So yeah,"}, {"start": 620.2399999999999, "end": 626.16, "text": " just keep that in mind, setting seeds in RL agent is very important. So you can actually reproduce"}, {"start": 626.16, "end": 634.7199999999999, "text": " results and have some stable trends. Linear schedule, simple class, which helps me anneal"}, {"start": 634.72, "end": 642.0, "text": " the epsilon hyperparameter from one to 01 over the span of certain number of steps. And that's"}, {"start": 642.0, "end": 648.8000000000001, "text": " just the thing that the paper used. So if I go to linear schedule, you can see it here. And the"}, {"start": 648.8000000000001, "end": 653.44, "text": " reason I'm actually going here is because I want to show you what I usually do when I develop a"}, {"start": 653.44, "end": 659.44, "text": " project. So you see the class here. And in order to be sure that at least some components are"}, {"start": 659.44, "end": 665.0400000000001, "text": " working, I create these subtests in the file itself. So you can see here, I actually instantiate"}, {"start": 665.0400000000001, "end": 671.6800000000001, "text": " those linear schedule and I test it. So if I run the utils, you'll see that, yeah, we have correct"}, {"start": 671.6800000000001, "end": 678.24, "text": " results. So it goes from one to 01 over the span of 50 steps. And the reason it's 50 is because"}, {"start": 678.24, "end": 683.5200000000001, "text": " I've set 50 here. So that's the way I test components. So first, most of the components"}, {"start": 683.5200000000001, "end": 689.2, "text": " are usually orthogonal, you can test them. And then once you combine them in the complete agent,"}, {"start": 689.2, "end": 695.9200000000001, "text": " infrastructure, you want to be confident in some components so that you can debug. Yeah. And even"}, {"start": 695.9200000000001, "end": 703.0400000000001, "text": " if, as you can see, even though I've done this, I'm still having problems debugging the RL agent. So"}, {"start": 703.6800000000001, "end": 708.1600000000001, "text": " testing is very important and just be pedantic when you're developing RL code."}, {"start": 709.0400000000001, "end": 716.72, "text": " Okay. Basic stuff here, PyTorch. If I have GPU and I have RTX 2080, I just fetch here the device."}, {"start": 716.72, "end": 723.9200000000001, "text": " I instantiate the DQN network, the target DQN, which we use to set the TDD targets in the DQN"}, {"start": 723.9200000000001, "end": 730.08, "text": " algorithm. And finally, this is the main thing, this ActorLearner class is what I used, is a nice"}, {"start": 730.08, "end": 736.08, "text": " abstraction and you'll see how it looks like. But basically you can see the shortest training loop"}, {"start": 736.08, "end": 743.9200000000001, "text": " ever here. And it's pretty intuitive. So basically until you have, so get experience count, so until"}, {"start": 743.92, "end": 752.24, "text": " you have collected this number of experiences, so frames, we just keep on collecting experience"}, {"start": 752.24, "end": 757.28, "text": " and we keep on learning. So first we go and collect experience. So we'll have an inner loop"}, {"start": 757.28, "end": 761.68, "text": " of four steps where we are collecting, interacting with the environment, taking the rewards, the"}, {"start": 761.68, "end": 766.4799999999999, "text": " down flags, the frames, observations, and we'll be storing them in a replay buffer. And once we"}, {"start": 766.4799999999999, "end": 772.9599999999999, "text": " have enough data, and there is this delay here, we start learning. So basically how DQN works is"}, {"start": 772.96, "end": 777.44, "text": " you initially don't learn, you're just collecting data and filling the replay buffer. And once you"}, {"start": 777.44, "end": 782.08, "text": " get to this point, now I start learning from the experience. And here I'm just sampling from the"}, {"start": 782.08, "end": 790.1600000000001, "text": " buffer and doing the HuberLoss stuff. Okay, let me continue. So along the way, I do note some stuff"}, {"start": 790.1600000000001, "end": 794.5600000000001, "text": " that's weird, that are weird, and some stuff that I want to do. So go through the code once more and"}, {"start": 794.5600000000001, "end": 799.12, "text": " make sure there are no bugs, it's an obvious one. But like this thing is something that I'm still"}, {"start": 799.12, "end": 805.12, "text": " confused about. For some reason, the game of Pong, even though you basically only need three actions,"}, {"start": 805.12, "end": 812.16, "text": " that's move up, move down, or no up, or basically just stay in place. For some reason, this"}, {"start": 812.16, "end": 818.08, "text": " Atari learning environment, the ALE, is giving me six outputs. If somebody knows it already,"}, {"start": 818.08, "end": 823.92, "text": " just write it down in the comments. But yeah, so that's how I think. So once I see"}, {"start": 823.92, "end": 830.16, "text": " something that's problematic, I just write it down in the code itself, using these to-dos. And this"}, {"start": 830.16, "end": 836.4, "text": " ID, the byte arm, basically highlights them in this yellow color. So it's really easy for me to"}, {"start": 836.4, "end": 842.16, "text": " organize them. And I also have the to-do list here, so I can just organize my to-dos really nicely."}, {"start": 842.16, "end": 848.88, "text": " So that's cool. Having said that, let's go to this ActualLearner class. Let me show you, and again,"}, {"start": 848.88, "end": 853.5999999999999, "text": " so a bunch of to-dos here, as you can see. And so ActualLearner, the init function is just basically"}, {"start": 853.6, "end": 860.16, "text": " a wrapper class. So I just store all of these replay buffers, dqns, etc., as like these"}, {"start": 862.08, "end": 868.96, "text": " private variables in Python. And what I do here is I just log the initial time of the program,"}, {"start": 868.96, "end": 874.32, "text": " because I'll be using this to calculate the frame per second metric. And yeah, so basically,"}, {"start": 874.32, "end": 884.08, "text": " just a container, the init function just takes care of these standard stuff. I just instantiate"}, {"start": 884.08, "end": 891.36, "text": " the loss here. I create the atom optimizer. And as you can see here, so the original paper used RMS"}, {"start": 891.36, "end": 897.36, "text": " prop, but that was before Atom was published, so I guess that's why they didn't use Atom. So I just,"}, {"start": 897.36, "end": 903.44, "text": " I'm using Atom, and if I'm having problems because of Atom, I'll try out RMS prop as well. And yeah,"}, {"start": 903.44, "end": 910.72, "text": " a short note to myself just to kind of freshen up my knowledge of how those algorithms exactly work,"}, {"start": 910.72, "end": 919.2, "text": " so that I can better debug this code. Okay, so first things first, collect experience. So that's"}, {"start": 919.2, "end": 925.6800000000001, "text": " one of the main functions. And as you can see here, so this number is equals four, so we just"}, {"start": 926.32, "end": 933.0400000000001, "text": " collect the experiences four times, so four steps in the environment. And what we do here, we just"}, {"start": 933.04, "end": 938.24, "text": " store the frame in the buffer, then we fetch the last observation, which will concatenate the four"}, {"start": 938.24, "end": 944.8, "text": " last frames. And if it doesn't have enough data, it will just fill in the black frames before. So"}, {"start": 944.8, "end": 949.52, "text": " yeah, basically, at the end, you'll have four frames, and some of them may be black if we don't"}, {"start": 949.52, "end": 955.4399999999999, "text": " have enough data. Then we sample the action using that observation. And that function just wraps up"}, {"start": 955.4399999999999, "end": 961.92, "text": " a couple of details from the DQN algorithm. It's a neat way to be concise here. And what I do is,"}, {"start": 961.92, "end": 966.64, "text": " initially, before the learning starts, and if you remember this part, this is the same thing as"}, {"start": 968.16, "end": 974.64, "text": " this part here. So we wait for 50k steps, and then we start learning. And before we start learning,"}, {"start": 974.64, "end": 978.64, "text": " we have, we just take random actions from the environment. So that means we'll be"}, {"start": 979.76, "end": 985.12, "text": " acting totally randomly in the environment initially. Then once that, once we have enough"}, {"start": 985.12, "end": 991.52, "text": " steps, we'll start doing something smart. And that's actually exploiting DQN's Epsilon greedy,"}, {"start": 991.52, "end": 997.12, "text": " as you can see here. So with Epsilon probability, so we just roll the dice with Epsilon probability,"}, {"start": 997.12, "end": 1003.68, "text": " we randomly act. Otherwise, we act greedily. We take the dead action that corresponds to the"}, {"start": 1003.68, "end": 1011.84, "text": " Q value that's the highest. And yeah, that's it. Let me get back to the algorithm sample action."}, {"start": 1011.84, "end": 1015.84, "text": " So we pick the action, we send the action to the environment, and that's your classical"}, {"start": 1015.84, "end": 1022.48, "text": " RL feedback loop. And I get the new frame back, the rewards and the dumb flag in case the episode"}, {"start": 1022.48, "end": 1030.56, "text": " is done or I've lost the life. Then what I do, I store effects. So basically, if you remember,"}, {"start": 1030.56, "end": 1035.2, "text": " I first stored a frame, but only now do I have the action and the reward and the dumb flags."}, {"start": 1035.2, "end": 1041.2, "text": " So at the same index, I just add that data so that we can later fetch it during the training,"}, {"start": 1041.2, "end": 1047.92, "text": " during the training when we are sampling from the buffer. Simple stuff here. And yeah,"}, {"start": 1047.92, "end": 1052.56, "text": " I'm logging some stuff here, as you can see. So some of the logging utilities are logging"}, {"start": 1053.6000000000001, "end": 1060.0, "text": " depending on the like every fourth episode. And some logging is just coupled to the number"}, {"start": 1060.0, "end": 1063.92, "text": " of steps and not to the number of episodes because it makes more sense and you'll see why."}, {"start": 1065.04, "end": 1069.52, "text": " Okay, the second important function is learning from experience. And you can see it here,"}, {"start": 1069.52, "end": 1075.12, "text": " basically, what I do here is I fetch from the replay buffer the batch size, which is 32 in the"}, {"start": 1075.12, "end": 1081.2, "text": " algorithm. And so we have all of we get the data as you can see observations, actions, rewards,"}, {"start": 1081.2, "end": 1088.48, "text": " next observations and dumb flags. And we just create the basically the target values, as you"}, {"start": 1088.48, "end": 1093.92, "text": " can see here, so it packs the next observation into the target DQN, I find the max values,"}, {"start": 1093.92, "end": 1100.16, "text": " max values, and we just add the rewards and scale with the discount factor gamma, we scale those"}, {"start": 1100.16, "end": 1107.92, "text": " numbers. And this thing here may be interesting. So basically, when the dumb flag is one, you want"}, {"start": 1107.92, "end": 1114.0800000000002, "text": " to make sure that this will zero out and this will be zero when the dumb flag is one, and we'll just"}, {"start": 1114.0800000000002, "end": 1119.68, "text": " have the reward. Why is that? Well, because basically, you don't want to have the you don't"}, {"start": 1119.68, "end": 1125.76, "text": " want to pass the final state, the terminal state into DQN, we just have the reward, we don't want"}, {"start": 1125.76, "end": 1130.4, "text": " to introduce the noise there because the state is already done. Basically, what the Q learning"}, {"start": 1130.4, "end": 1136.24, "text": " function does, it predicts the expected reward. And since the state is already since the game is"}, {"start": 1136.24, "end": 1143.52, "text": " done, the expected reward should be zero. And yeah, that's why we zero out that part. Then we have"}, {"start": 1143.52, "end": 1150.8, "text": " the observations, we pass them through DQN. And we just take the actions that we sampled during the"}, {"start": 1151.6, "end": 1157.44, "text": " experience collection. And we take those Q values. And finally, we just have the loss between the"}, {"start": 1157.44, "end": 1162.24, "text": " target values. And so that's your DQN algorithm. And this part is not that important if you didn't"}, {"start": 1162.24, "end": 1166.32, "text": " understand. I'm just walking you through how I'm thinking and how I'm structuring the code and"}, {"start": 1166.32, "end": 1172.48, "text": " everything. Okay, then basic stuff. We're zeroing the gradient gradients, we do the backward, which"}, {"start": 1172.48, "end": 1178.64, "text": " will accumulate the grad fields in PyTorch. So every single variable will have the grad fields"}, {"start": 1178.64, "end": 1185.3600000000001, "text": " filled in with the gradient. And we do the step. So that's it. That's how we learn. Two more details."}, {"start": 1186.32, "end": 1195.6, "text": " First off, every fourth update, so no, actually every 10k update, I'll be doing a hard"}, {"start": 1195.6, "end": 1202.9599999999998, "text": " update of the target network. And that's what the original paper showed, improved the stability"}, {"start": 1202.9599999999998, "end": 1207.4399999999998, "text": " significantly. So aside from replay buffer, this thing is very important in order to have an"}, {"start": 1207.4399999999998, "end": 1214.24, "text": " algorithm that's stable and that can work. So finally, you can see I'm logging gradients here."}, {"start": 1214.24, "end": 1218.7199999999998, "text": " And that's super important because I'm still trying to debug this thing. And yeah, okay, so I think"}, {"start": 1218.72, "end": 1225.68, "text": " I've got enough data in the tensor board. So I'll just go ahead and stop this run. And what I'll do,"}, {"start": 1225.68, "end": 1230.64, "text": " I want to show you what the visual observation is. Basically, collect experience, I'll just"}, {"start": 1230.64, "end": 1237.52, "text": " uncomment this thing here. I'll put a breaking point there and I'll rerun this thing. So let me"}, {"start": 1237.52, "end": 1245.28, "text": " show you what this does. Whoops, that's the wrong file. I'm starting the main file. Okay, if I go"}, {"start": 1245.28, "end": 1251.28, "text": " here and we just plot the function, you can see initially because we only have one frame,"}, {"start": 1252.08, "end": 1257.6, "text": " and this is the observation. So the last three frames will be black. Okay, so if we go and do"}, {"start": 1257.6, "end": 1263.6, "text": " another step, so as you can see, we did another step. I step into here, I plotted, we now have"}, {"start": 1263.6, "end": 1270.72, "text": " two frames. So if I do another step again, we did another step and I step in, you can see we have"}, {"start": 1270.72, "end": 1277.68, "text": " three frames. So that's what I have this for. So just a simple function which will help me"}, {"start": 1277.68, "end": 1282.32, "text": " better understand what's going on. And okay, let me show you the logging functions in tensor"}, {"start": 1282.32, "end": 1286.88, "text": " board and we're pretty much done. When it comes to logging, I'm just, I'm basically logging the"}, {"start": 1286.88, "end": 1291.52, "text": " stuff you already saw in the tensor board. So reward per episode, blah, blah, blah, nothing smart"}, {"start": 1291.52, "end": 1295.92, "text": " there. So maybe the only a bit more interesting part is here. So I'm iterating through the"}, {"start": 1295.92, "end": 1300.88, "text": " parameters of the decan model and I'm just calculating the, as you can see, I take the"}, {"start": 1300.88, "end": 1307.6000000000001, "text": " gradient, the data, and I did the norm, the L2 norm, P equals two L2 norm, and I just"}, {"start": 1308.5600000000002, "end": 1315.04, "text": " basically log those numbers. And finally, I'm accumulating these numbers in the total grad"}, {"start": 1315.04, "end": 1320.48, "text": " so we can have the complete vector and not just the parameters for a specific layer, but for the"}, {"start": 1320.48, "end": 1326.4, "text": " whole network itself. Okay, let me show you the tensor board finally and see what the data we've"}, {"start": 1326.4, "end": 1332.8, "text": " got. Okay, Epsilon as expected, just linearly annealed to 0.1. That's not so interesting."}, {"start": 1332.8, "end": 1340.32, "text": " FPS, as you can see, it's pretty stable, around 330 frames. I don't know whether that's good or"}, {"start": 1340.32, "end": 1346.24, "text": " bad because I haven't been experimenting enough, but we'll see. That sounds like a nice number."}, {"start": 1346.24, "end": 1353.76, "text": " Huber loss, as you can see, it's just random noise pretty much. I don't see any downward trends, so"}, {"start": 1353.76, "end": 1359.36, "text": " something's wrong, or maybe I just didn't have enough steps. That's also a possibility. Also"}, {"start": 1359.36, "end": 1366.08, "text": " rewards, as you can see, I'm almost constantly at minus 21. That means I basically, the agent loses"}, {"start": 1366.08, "end": 1372.48, "text": " all of the 21 games of Pong against the built-in agent, so that means it's just playing really bad."}, {"start": 1372.48, "end": 1377.04, "text": " Steps per episode, again, really noisy, and you can see the number of steps that's needed per"}, {"start": 1377.04, "end": 1383.6, "text": " episode, so that's around maybe the mean is around 850 or something. And finally, we can see the"}, {"start": 1384.24, "end": 1392.16, "text": " grads, and they are completely zero. So if you've seen the bug in my code, comment down in the"}, {"start": 1392.16, "end": 1396.96, "text": " comment section. I'll hopefully have this thing running in a couple of days, and then I'll need"}, {"start": 1396.96, "end": 1402.96, "text": " a couple more days to just polish everything, create a readme, so that people can kind of use"}, {"start": 1402.96, "end": 1409.2, "text": " it and not experience the problems I did. So yeah, one more thing I want to mention, which is really"}, {"start": 1409.2, "end": 1416.0, "text": " important, how I actually go about developing these projects. So usually, unless nobody has"}, {"start": 1416.0, "end": 1421.44, "text": " published any code whatsoever, which I did have a couple of occasions where I had to implement a"}, {"start": 1421.44, "end": 1426.24, "text": " paper totally from scratch, I just had the paper, so what I ended up doing is pinning the entire"}, {"start": 1426.24, "end": 1432.08, "text": " authors and communicating with them, so I did get some snippets there. So basically, you always want"}, {"start": 1432.08, "end": 1436.96, "text": " to start with some code and understand what other people are doing. So I did investigate at least"}, {"start": 1436.96, "end": 1443.04, "text": " five, six, or seven, something like that, PyTorch implementations, I think six. And one of them here"}, {"start": 1443.04, "end": 1448.88, "text": " is stable baselines, so I was just stepping through the code, looking what those folks did, and as I"}, {"start": 1448.88, "end": 1453.68, "text": " said, they didn't have the smart replay buffers, so I had to implement the buffer from scratch,"}, {"start": 1453.68, "end": 1460.96, "text": " and stuff like that. But I did learn some nice tricks from this repo, so that's a cool thing to"}, {"start": 1460.96, "end": 1465.68, "text": " do when you're developing a project. Go ahead and first analyze the other projects, and you'll learn"}, {"start": 1465.68, "end": 1470.88, "text": " some stuff additionally, and it will be a much more rewarding experience. You'll learn more if you also"}, {"start": 1470.88, "end": 1477.92, "text": " understand what other people have been doing. Okay, having said that, maybe the last thing I want to"}, {"start": 1477.92, "end": 1484.5600000000002, "text": " show you are quickly the DQN model, what I've done there. And you can see I just created a super"}, {"start": 1484.5600000000002, "end": 1492.48, "text": " generic model there, so you can play with it, and you'll have the code in a couple of days, hopefully"}, {"start": 1492.48, "end": 1497.92, "text": " until next weekend, actually. And you can see I have this automatic part here, where because, as"}, {"start": 1497.92, "end": 1502.8000000000002, "text": " you know, we have three convolutional layers, we have fully connected layer, that means that"}, {"start": 1502.8, "end": 1508.08, "text": " basically you can't change the input, because otherwise the compatibility between the volume"}, {"start": 1508.08, "end": 1512.96, "text": " that comes out from the third comb layer won't be compatible with the first fully connected layer,"}, {"start": 1512.96, "end": 1519.12, "text": " so you either have to keep the observation constant, or you can do it like me here."}, {"start": 1519.12, "end": 1526.8799999999999, "text": " So I'm basically passing the input volume, and I can automatically calculate the number of neurons"}, {"start": 1526.8799999999999, "end": 1531.2, "text": " that I need in the first fully connected layer. So that's this part here."}, {"start": 1531.2, "end": 1536.72, "text": " And yeah, that's not an interesting replay buffer, on the other hand. As you can see,"}, {"start": 1536.72, "end": 1546.48, "text": " the main thing I've done it is because I only need 7GB of memory, only of RAM, instead of 28 for the"}, {"start": 1546.48, "end": 1555.92, "text": " implementation that these stable baselines have. And yeah, basically a couple of public functions"}, {"start": 1555.92, "end": 1561.92, "text": " here, and the bunch of work happens here. There is not a lot of code, but there are certain edge"}, {"start": 1561.92, "end": 1567.8400000000001, "text": " cases, and the implementation I actually took an inspiration from was this Berkeley's implementation,"}, {"start": 1567.8400000000001, "end": 1574.8000000000002, "text": " and they also have a small bug. It's a subtle one, it won't actually impact the final result of the"}, {"start": 1574.8000000000002, "end": 1580.5600000000002, "text": " Dequeon agent, but nonetheless I did found it and submit an issue. And yeah, that's pretty much it."}, {"start": 1580.56, "end": 1586.56, "text": " Hopefully you found this video useful. If you did, leave a comment, subscribe, share the video,"}, {"start": 1586.56, "end": 1614.56, "text": " and I hope to see you next time."}]