Feed-forward method (training) | Neural Style Transfer #6
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The 6th video in the NST series! 🎨

You'll learn about:
✔️ how to train and monitor your NST feed-forward neural network

Note: you'll need a decent machine to train these yourself in a reasonable amount of time.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ GitHub code: https://github.com/gordicaleksa/pytorch-nst-feedforward
✅ Paper (Johnson et al.): https://arxiv.org/pdf/1603.08155.pdf
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
0:00 - Quick training script walk-through
3:15 - Setting up training parameters
4:38 - MS Coco download
5:15 - Running the training procedure & Tensorboard
8:38 - MS COCO quick walk-through
9:30 - Saving models (checkpoints + binaries)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neuralstyletransfer #deeplearning #ai
 So in the last video we saw how to run stylization scripts and in this feedforward neural style transfer method. And in this one we're going to see how to run trainings and how to leverage tensorboard in order to track your experiments. Okay, so go ahead and open the training script. Let's jump to this function. So this is just some boilerplate code. And then we just create a PyTorch data loader which will basically just take images from MSCoco dataset, resize them to 256 by 256, and concatenate them into batches and also do image normalization. So afterwards we have this transformer network which is the meat of this project basically because that's the network that we'll be training. Perceptual loss net is we're using VGG16 here. It's a usual thing to do in an NST field. And yeah, just add an optimizer, nothing special there. And we're going to load this style image and you can choose between basically whatever you put in this style images directory. You'll be able to use that one to train the model for that specific style image. What we do here is we just pass the style image through the perceptual net and we just find the style representation which is just a collection of gram matrices. I just print the header here, nothing special there, just boilerplate code. And this is the main training loop. So what we do here is we, as I said, we have this train loader. So we just take one batch and we pass the batch through the transformer net. And whatever comes out is what we want this stylized batch to have the same content as the content image. And to have the same style as the style image we just initialized a couple of seconds ago. So we take this content batch which is just a collection of images from Ms. Coco. We just pass it through the perceptual loss net and we also pass this stylized batch that just came out of the transformer net through the VGG. So bottom line, this target content representation is going to be different for every single iteration. Whereas the style representation is going to be the same because these models are kind of not flexible. They can only be trained for a single style. We just calculate content loss and adjust MSC loss between the target content representation I just mentioned and between the current content representation. We just find the style loss by also doing MSC on gram matrices. And TV loss is for kind of enforces image smoothness. And in my experience, it did not help as much. So I've set it to zero but you can play with it if you get some really strange noise in the output image of your model. And then we just combine the thought loss as a sum of those three components. Let's see how we can train these models. So just go ahead down here and first thing you do is you pick your style image. So you pick a style image that you want to train your model on. So let's set, for example, mosaic here and you can pick whatever options you have here. You can just play some new style image in this directory style images and you can use that style image. So afterwards, you the only parameter pretty much that you'll have to change and tweak around is this style weight parameter. And it's for e to the power of five at the moment. So let's just stick with it. And you'll probably have to experiment with with e-books. So, for example, you can use only one e-book and put maybe three thirty images here instead of none, which will use all of the MSCoco images. And that's around eighty three thousand images output, maybe at only two thousand here just for demonstration sake. And I'm going to put only one e-book here. TensorBoard is set to true. We want to enable this. This will make your life easier when monitoring these experiments. So we're just going to head and so beforehand, actually, we got a we got a download MSCoco data set. And I just you just basically open this resource downloader. I've already prepared it for you. So you just have to to place the download choice here to one. And you will go ahead and download the MSCoco. You just go ahead and run this and it will start downloading the MSCoco data set. So I've already done that. So I'm just going to stop it here. And it's a big data set. I think it's around 13 gigabytes. So just keep that on the back of your mind. And now we're just going to run the training script and it will start training the model. I've learned to train and actually I've set the number of data points to twenty thousand so that you can better see the curves in TensorBoard. So now let's just open the TensorBoard. You can see the command here in the console. You just need to open your your account, the prompt and navigate to project route, activate the environment. And then you can just copy paste the command in the console. So TensorBoard, log, dear, blah, blah. Copy that. Paste it here. Run it. And you'll see the URL here. It's up. It will just open like a local local server which you can access through your browser. Just paste it here and you'll see the TensorBoard. Once it opens, you just go and scroll to your experiment and it will be on the bottom of this list. You just activate the five of these. Basically, I'm going to explain what these mean. So content loss and the style loss are the pretty much the most important ones. And you can see that the content loss, even after five thousand batches, i.e. twenty thousand images, is still lowering down. So you can train this model for at least at least twenty thousand more. The important thing with style loss is that it doesn't have huge spikes. Otherwise, something went wrong and you probably need to lower down your style weight and then your model will be able to train. TV loss is zero because I've set the weight to zero. You'll only need to use this one if you get some really unsmooth output from your model. And this bottom curve here just tells me some certain statistics of the stylized batch that goes. That's the output from the transformer transformer net I showed you in the code in the beginning of the video. These actually make a lot of sense because the input imagery that goes through the transformer net is ImageNet normalized. So it's kind of centralized around zero and the output should be similar, should have a similar distribution. So we can see that here. And aside from these curves, I also have some imagery logged during the training. And you can see how your your model is learning to better reproduce the style of the image you're training it for. So in the beginning is just some random junk imagery. And then as you go, as the time passes by and as the model have seen more and more data points or images from Ms. Coco in this in this case, it just gets better and better. And you can see that the content is getting more prominent and stylization is getting better. This is really useful to just debug and see if your model is actually learning the stylized imagery so that you don't have to wait for two hours and only then figure out that it's not learning anything. And one thing to note, you'll probably need a really strong machine to actually train this locally. As you can see, the ventilators on my machine are pretty working and pretty heavy. So just keep that in mind. I'm going to close the answer board now and just going to lead you through the Ms. Coco data set. Just show you the imagery here. It's always a good idea to go through the data set and just check out images you're training on just manually and kind of get the feeling of how it looks like. And you can see the imagery here on the Ms. Coco just random, random stuff pretty much. And it actually doesn't matter because the model doesn't care which images you pass it in. You only care to set some nice style style image and it will iterate through these content images and learn how to lower both the content as well as the style loss. And you can see the imagery here. Finally, you have your model saved into both binaries and checkpoints. So checkpoints basically serves to save intermediate models while your model is actually being trained. And you'll find it under the subdirectory with the same name as your style image. So our checkpoints will be under mosaic here because we use the mosaic style image. But the final model will be actually placed here in the binaries directory. And you'll be able to take that model and run it through the stylization script that I showed you in the last video. So that was it for this video. You'll learn how the training loop looks like. You learn how to start your own training, how to monitor the trainings using tensor boards, curves and and imagery, logged imagery. And you also know where to find your binaries that you can run later on in the stylization script. Thanks for watching. If you found this video useful, consider subscribing and sharing and see you in the next video.
[{"start": 0.0, "end": 6.8, "text": " So in the last video we saw how to run stylization scripts and in this feedforward neural style transfer method."}, {"start": 6.8, "end": 13.4, "text": " And in this one we're going to see how to run trainings and how to leverage tensorboard in order to track your experiments."}, {"start": 13.4, "end": 16.4, "text": " Okay, so go ahead and open the training script."}, {"start": 16.4, "end": 18.7, "text": " Let's jump to this function."}, {"start": 18.7, "end": 22.1, "text": " So this is just some boilerplate code."}, {"start": 22.1, "end": 32.6, "text": " And then we just create a PyTorch data loader which will basically just take images from MSCoco dataset, resize them to 256 by 256,"}, {"start": 32.6, "end": 37.6, "text": " and concatenate them into batches and also do image normalization."}, {"start": 37.6, "end": 47.6, "text": " So afterwards we have this transformer network which is the meat of this project basically because that's the network that we'll be training."}, {"start": 47.6, "end": 52.300000000000004, "text": " Perceptual loss net is we're using VGG16 here."}, {"start": 52.300000000000004, "end": 55.8, "text": " It's a usual thing to do in an NST field."}, {"start": 55.8, "end": 60.800000000000004, "text": " And yeah, just add an optimizer, nothing special there."}, {"start": 60.800000000000004, "end": 71.5, "text": " And we're going to load this style image and you can choose between basically whatever you put in this style images directory."}, {"start": 71.5, "end": 75.6, "text": " You'll be able to use that one to train the model for that specific style image."}, {"start": 75.6, "end": 87.6, "text": " What we do here is we just pass the style image through the perceptual net and we just find the style representation which is just a collection of gram matrices."}, {"start": 87.6, "end": 91.89999999999999, "text": " I just print the header here, nothing special there, just boilerplate code."}, {"start": 91.89999999999999, "end": 95.1, "text": " And this is the main training loop."}, {"start": 95.1, "end": 99.6, "text": " So what we do here is we, as I said, we have this train loader."}, {"start": 99.6, "end": 106.1, "text": " So we just take one batch and we pass the batch through the transformer net."}, {"start": 106.1, "end": 116.1, "text": " And whatever comes out is what we want this stylized batch to have the same content as the content image."}, {"start": 116.1, "end": 122.6, "text": " And to have the same style as the style image we just initialized a couple of seconds ago."}, {"start": 122.6, "end": 128.6, "text": " So we take this content batch which is just a collection of images from Ms. Coco."}, {"start": 128.6, "end": 136.1, "text": " We just pass it through the perceptual loss net and we also pass this stylized batch that just came out of the transformer net through the VGG."}, {"start": 136.1, "end": 143.6, "text": " So bottom line, this target content representation is going to be different for every single iteration."}, {"start": 143.6, "end": 150.6, "text": " Whereas the style representation is going to be the same because these models are kind of not flexible."}, {"start": 150.6, "end": 152.6, "text": " They can only be trained for a single style."}, {"start": 152.6, "end": 163.6, "text": " We just calculate content loss and adjust MSC loss between the target content representation I just mentioned and between the current content representation."}, {"start": 163.6, "end": 169.6, "text": " We just find the style loss by also doing MSC on gram matrices."}, {"start": 169.6, "end": 175.1, "text": " And TV loss is for kind of enforces image smoothness."}, {"start": 175.1, "end": 179.6, "text": " And in my experience, it did not help as much."}, {"start": 179.6, "end": 187.6, "text": " So I've set it to zero but you can play with it if you get some really strange noise in the output image of your model."}, {"start": 187.6, "end": 194.6, "text": " And then we just combine the thought loss as a sum of those three components."}, {"start": 194.6, "end": 197.6, "text": " Let's see how we can train these models."}, {"start": 197.6, "end": 203.1, "text": " So just go ahead down here and first thing you do is you pick your style image."}, {"start": 203.1, "end": 207.6, "text": " So you pick a style image that you want to train your model on."}, {"start": 207.6, "end": 215.6, "text": " So let's set, for example, mosaic here and you can pick whatever options you have here."}, {"start": 215.6, "end": 221.6, "text": " You can just play some new style image in this directory style images and you can use that style image."}, {"start": 221.6, "end": 230.6, "text": " So afterwards, you the only parameter pretty much that you'll have to change and tweak around is this style weight parameter."}, {"start": 230.6, "end": 235.6, "text": " And it's for e to the power of five at the moment."}, {"start": 235.6, "end": 238.6, "text": " So let's just stick with it."}, {"start": 238.6, "end": 244.6, "text": " And you'll probably have to experiment with with e-books."}, {"start": 244.6, "end": 252.6, "text": " So, for example, you can use only one e-book and put maybe three thirty images here instead of none,"}, {"start": 252.6, "end": 255.6, "text": " which will use all of the MSCoco images."}, {"start": 255.6, "end": 263.6, "text": " And that's around eighty three thousand images output, maybe at only two thousand here just for demonstration sake."}, {"start": 263.6, "end": 267.6, "text": " And I'm going to put only one e-book here."}, {"start": 267.6, "end": 270.6, "text": " TensorBoard is set to true. We want to enable this."}, {"start": 270.6, "end": 275.6, "text": " This will make your life easier when monitoring these experiments."}, {"start": 275.6, "end": 283.6, "text": " So we're just going to head and so beforehand, actually, we got a we got a download MSCoco data set."}, {"start": 283.6, "end": 286.6, "text": " And I just you just basically open this resource downloader."}, {"start": 286.6, "end": 294.6, "text": " I've already prepared it for you. So you just have to to place the download choice here to one."}, {"start": 294.6, "end": 298.6, "text": " And you will go ahead and download the MSCoco."}, {"start": 298.6, "end": 304.6, "text": " You just go ahead and run this and it will start downloading the MSCoco data set."}, {"start": 304.6, "end": 307.6, "text": " So I've already done that. So I'm just going to stop it here."}, {"start": 307.6, "end": 312.6, "text": " And it's a big data set. I think it's around 13 gigabytes."}, {"start": 312.6, "end": 314.6, "text": " So just keep that on the back of your mind."}, {"start": 314.6, "end": 321.6, "text": " And now we're just going to run the training script and it will start training the model."}, {"start": 321.6, "end": 328.6, "text": " I've learned to train and actually I've set the number of data points to twenty thousand so that you can better see the curves in TensorBoard."}, {"start": 328.6, "end": 335.6, "text": " So now let's just open the TensorBoard. You can see the command here in the console."}, {"start": 335.6, "end": 342.6, "text": " You just need to open your your account, the prompt and navigate to project route, activate the environment."}, {"start": 342.6, "end": 346.6, "text": " And then you can just copy paste the command in the console."}, {"start": 346.6, "end": 352.6, "text": " So TensorBoard, log, dear, blah, blah. Copy that."}, {"start": 352.6, "end": 356.6, "text": " Paste it here. Run it."}, {"start": 356.6, "end": 360.6, "text": " And you'll see the URL here. It's up."}, {"start": 360.6, "end": 367.6, "text": " It will just open like a local local server which you can access through your browser."}, {"start": 367.6, "end": 371.6, "text": " Just paste it here and you'll see the TensorBoard."}, {"start": 371.6, "end": 380.6, "text": " Once it opens, you just go and scroll to your experiment and it will be on the bottom of this list."}, {"start": 380.6, "end": 382.6, "text": " You just activate the five of these."}, {"start": 382.6, "end": 384.6, "text": " Basically, I'm going to explain what these mean."}, {"start": 384.6, "end": 389.6, "text": " So content loss and the style loss are the pretty much the most important ones."}, {"start": 389.6, "end": 397.6, "text": " And you can see that the content loss, even after five thousand batches, i.e. twenty thousand images, is still lowering down."}, {"start": 397.6, "end": 404.6, "text": " So you can train this model for at least at least twenty thousand more."}, {"start": 404.6, "end": 409.6, "text": " The important thing with style loss is that it doesn't have huge spikes."}, {"start": 409.6, "end": 416.6, "text": " Otherwise, something went wrong and you probably need to lower down your style weight and then your model will be able to train."}, {"start": 416.6, "end": 420.6, "text": " TV loss is zero because I've set the weight to zero."}, {"start": 420.6, "end": 426.6, "text": " You'll only need to use this one if you get some really unsmooth output from your model."}, {"start": 426.6, "end": 433.6, "text": " And this bottom curve here just tells me some certain statistics of the stylized batch that goes."}, {"start": 433.6, "end": 438.6, "text": " That's the output from the transformer transformer net I showed you in the code in the beginning of the video."}, {"start": 438.6, "end": 445.6, "text": " These actually make a lot of sense because the input imagery that goes through the transformer net is ImageNet normalized."}, {"start": 445.6, "end": 453.6, "text": " So it's kind of centralized around zero and the output should be similar, should have a similar distribution."}, {"start": 453.6, "end": 454.6, "text": " So we can see that here."}, {"start": 454.6, "end": 460.6, "text": " And aside from these curves, I also have some imagery logged during the training."}, {"start": 460.6, "end": 467.6, "text": " And you can see how your your model is learning to better reproduce the style of the image you're training it for."}, {"start": 467.6, "end": 472.6, "text": " So in the beginning is just some random junk imagery."}, {"start": 472.6, "end": 483.6, "text": " And then as you go, as the time passes by and as the model have seen more and more data points or images from Ms. Coco in this in this case,"}, {"start": 483.6, "end": 485.6, "text": " it just gets better and better."}, {"start": 485.6, "end": 493.6, "text": " And you can see that the content is getting more prominent and stylization is getting better."}, {"start": 493.6, "end": 505.6, "text": " This is really useful to just debug and see if your model is actually learning the stylized imagery so that you don't have to wait for two hours and only then figure out that it's not learning anything."}, {"start": 505.6, "end": 509.6, "text": " And one thing to note, you'll probably need a really strong machine to actually train this locally."}, {"start": 509.6, "end": 514.6, "text": " As you can see, the ventilators on my machine are pretty working and pretty heavy."}, {"start": 514.6, "end": 515.6, "text": " So just keep that in mind."}, {"start": 515.6, "end": 522.6, "text": " I'm going to close the answer board now and just going to lead you through the Ms. Coco data set."}, {"start": 522.6, "end": 525.6, "text": " Just show you the imagery here."}, {"start": 525.6, "end": 535.6, "text": " It's always a good idea to go through the data set and just check out images you're training on just manually and kind of get the feeling of how it looks like."}, {"start": 535.6, "end": 543.6, "text": " And you can see the imagery here on the Ms. Coco just random, random stuff pretty much."}, {"start": 543.6, "end": 551.6, "text": " And it actually doesn't matter because the model doesn't care which images you pass it in."}, {"start": 551.6, "end": 564.6, "text": " You only care to set some nice style style image and it will iterate through these content images and learn how to lower both the content as well as the style loss."}, {"start": 564.6, "end": 568.6, "text": " And you can see the imagery here."}, {"start": 568.6, "end": 574.6, "text": " Finally, you have your model saved into both binaries and checkpoints."}, {"start": 574.6, "end": 580.6, "text": " So checkpoints basically serves to save intermediate models while your model is actually being trained."}, {"start": 580.6, "end": 587.6, "text": " And you'll find it under the subdirectory with the same name as your style image."}, {"start": 587.6, "end": 592.6, "text": " So our checkpoints will be under mosaic here because we use the mosaic style image."}, {"start": 592.6, "end": 598.6, "text": " But the final model will be actually placed here in the binaries directory."}, {"start": 598.6, "end": 605.6, "text": " And you'll be able to take that model and run it through the stylization script that I showed you in the last video."}, {"start": 605.6, "end": 606.6, "text": " So that was it for this video."}, {"start": 606.6, "end": 609.6, "text": " You'll learn how the training loop looks like."}, {"start": 609.6, "end": 620.6, "text": " You learn how to start your own training, how to monitor the trainings using tensor boards, curves and and imagery, logged imagery."}, {"start": 620.6, "end": 628.6, "text": " And you also know where to find your binaries that you can run later on in the stylization script."}, {"start": 628.6, "end": 629.6, "text": " Thanks for watching."}, {"start": 629.6, "end": 651.6, "text": " If you found this video useful, consider subscribing and sharing and see you in the next video."}]