Feed-forward method | Neural Style Transfer #5
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The 5th video in the NST series! 🎨

You'll learn about:
✔️ fast, feed-forward, CNN-based neural style transfer!

You don't need to read the research paper to be able to play with the code and make awesome stuff! You'll probably need some basic understanding if you want to train the models yourself - but I'll cover that in the next video!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ GitHub code: https://github.com/gordicaleksa/pytorch-nst-feedforward
✅ Paper (Johnson et al.): https://arxiv.org/pdf/1603.08155.pdf
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
0:00 - GitHub repo walk-through
0:50 - Setup timelapse (check the repo for details)
1:03 - step1: Download pre-trained models
1:30 - step2: Run the stylization script
2:10 - Stylization script walk-through

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition,
rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neuralstyletransfer #deeplearning #ai
 Okay, so this video is going to be really nice and short. I went ahead and reconstructed the original Johnson's paper on fast neural style transfer. It's a CNN based approach as opposed to the GATIS' approach which was optimization based. Let's see some results I got using the four pre-trained models I trained beforehand. So in the left column you can see the style images I used and on the right you can see the output from the models I trained. So keep in mind that the the model in the top row is the only one that was fully trained. So it's all like the two epochs of the whole MSCoco dataset. So that's around 83 000 images whereas the three bottom ones still need to be additionally trained. Now I'm just going to do a quick time lapse of the setup because we already did that in the previous coding video. Once you have your con environment configured there are only two more steps to get some results. So the first one is go ahead and run this resource downloader script and it will just go ahead and download the pre-trained models that I've uploaded right here to this URL on Dropbox. You will then unzip them and place them in this folder here models-binaries. So you'll have four pre-trained models on your disposal. And the second step is just go ahead and run this stylization script. It's located here. Just run it with the default parameters and we'll get some results. This is the result we get using the default content image that's located here. Just go to data content images and this one is used by default. So I'm just going to go and show you a little bit about how the stylization script itself works. I'm going to go ahead and stop this one here. Oops and I'm just going to show you where it's actually saved. It saved to the output images directory here. So you'll be able to find it later on in this directory. Okay let's see how the script works. Let me close this one and zoom this in a little bit like this. So what we have here is a couple of default locations. First one is for content images and we already saw that one that's in here. We have output images where the images will get dumped. Bineries where we have our models. And this is just basic error checking here. Basically checking if we only have PyTorch models inside of this binaries directory. And I just create the output directory where we'll dump images. And here are the arguments. These are just the content image that you want to use, that you want to pass through the model. Then we have the, you want to set the width of the output image and it's set to 500 by default here. We have default model that's called Mosaic. And we just wrap all of those parameters and we call the Stylized Static Image function. Let's see what this function actually does. So if we go upside here we can first see, we first want to check out if the device has a GPU or only CPU support. And then we basically just create this image path. We pass it through this function called prepareImage which will kind of add a batch dimension and do some normalization. We'll check the function a bit later. And I just instantiate here the Transform model which is the actual model that performs the stylization. We figure out the state. So we just load the state from a PyTorch model that's in binaries here. We print some metadata and we finally load the weights inside of the the model and we just put it into this evolve mode which is really important if you want to do inference. You want to do this, you want to call this function here. And finally we just wrap this into this context torch no grad which will basically forbid PyTorch to calculate gradients which will, which would kind of be a huge memory overhead. So we call the model on the content image. We just paste it to CPU. We convert it to NumPy here. And because we had the batch dimension that we previously added we got to extract the 0th batch here. And that's the stylized image and we just save the image. That's the image we saw a couple of minutes ago that was dumped here into output images folder. This one. Let's just check out that prepare image we saw in the utils here. So this is how it looks like. It basically just loads the image from the specified path specified by this variable image path. And that will just basically create a NumPy image normalized to 0 to 1 range. And then we'll pretty much add the this normalization transform that PyTorch provides us with. And we'll use ImageNet's mean and standard deviation to normalize our images. After we apply the transformation here we just push the push this image onto the GPU if we have one. And we just add this batch dimension in front because that's because models are always expecting this batch dimension. So pretty simple stuff. In the next video I'll cover training that will be a bit more challenging. We're going to figure out how to use TensorBoard and just debug and visualize our training metrics. So let me just go ahead and try one more image here. So I'm going to use an image called figures and let me set the width here to 350 say. We'll use the same model mosaic. And if I start that one, if I run it, this is what we get. So how good the output image will be actually depends also on the size that you put here in the width. So if I put 550 say here and I run that one it will have different stylization. And it's much nicer if you ask me. In the next video we'll cover training until that time just go ahead and play with this repo. Try and figure out how the things work and that will be a really nice learning experience. If you like the content go ahead and subscribe, like and share the videos if you think they can bring value to somebody else. See you next time!
[{"start": 0.0, "end": 3.92, "text": " Okay, so this video is going to be really nice and short. I went ahead and"}, {"start": 3.92, "end": 8.32, "text": " reconstructed the original Johnson's paper on fast neural style transfer."}, {"start": 8.32, "end": 11.6, "text": " It's a CNN based approach as opposed to the"}, {"start": 11.6, "end": 14.64, "text": " GATIS' approach which was optimization based."}, {"start": 14.64, "end": 18.96, "text": " Let's see some results I got using the four pre-trained models I trained"}, {"start": 18.96, "end": 22.8, "text": " beforehand. So in the left column you can see the style images"}, {"start": 22.8, "end": 27.28, "text": " I used and on the right you can see the output from the models I trained."}, {"start": 27.28, "end": 30.880000000000003, "text": " So keep in mind that the the model in the top row is"}, {"start": 30.880000000000003, "end": 34.08, "text": " the only one that was fully trained. So it's all like"}, {"start": 34.08, "end": 38.24, "text": " the two epochs of the whole MSCoco dataset."}, {"start": 38.24, "end": 42.32, "text": " So that's around 83 000 images whereas the three"}, {"start": 42.32, "end": 45.52, "text": " bottom ones still need to be additionally trained. Now I'm just going"}, {"start": 45.52, "end": 48.08, "text": " to do a quick time lapse of the setup because we"}, {"start": 48.08, "end": 58.32, "text": " already did that in the previous coding video."}, {"start": 62.72, "end": 66.16, "text": " Once you have your con environment configured there are only two more steps"}, {"start": 66.16, "end": 70.4, "text": " to get some results. So the first one is go ahead and run this"}, {"start": 70.4, "end": 75.52, "text": " resource downloader script and it will just go ahead and download the pre-trained"}, {"start": 75.52, "end": 79.28, "text": " models that I've uploaded right here to this"}, {"start": 79.28, "end": 84.56, "text": " URL on Dropbox. You will then unzip them and place them in this folder"}, {"start": 84.56, "end": 88.24, "text": " here models-binaries. So you'll have four pre-trained"}, {"start": 88.24, "end": 91.52, "text": " models on your disposal. And the second step is just go ahead and run this"}, {"start": 91.52, "end": 96.0, "text": " stylization script. It's located here. Just run it with the"}, {"start": 96.0, "end": 98.24, "text": " default parameters and we'll get some results."}, {"start": 98.24, "end": 102.08, "text": " This is the result we get using the default content image that's"}, {"start": 102.08, "end": 108.16, "text": " located here. Just go to data content images and this one is used by"}, {"start": 108.16, "end": 112.88, "text": " default. So I'm just going to go and show you a"}, {"start": 112.88, "end": 115.75999999999999, "text": " little bit about how the stylization script itself works."}, {"start": 115.75999999999999, "end": 119.12, "text": " I'm going to go ahead and stop this one here."}, {"start": 119.12, "end": 124.08, "text": " Oops and I'm just going to show you where it's actually saved. It saved to the"}, {"start": 124.08, "end": 128.16, "text": " output images directory here. So you'll be able to find it later"}, {"start": 128.16, "end": 130.8, "text": " on in this directory. Okay let's see how the"}, {"start": 130.8, "end": 135.52, "text": " script works. Let me close this one and zoom this in a"}, {"start": 135.52, "end": 140.08, "text": " little bit like this. So what we have here is a"}, {"start": 140.08, "end": 144.4, "text": " couple of default locations. First one is for content images and we"}, {"start": 144.4, "end": 148.48000000000002, "text": " already saw that one that's in here. We have output images where the"}, {"start": 148.48000000000002, "end": 154.4, "text": " images will get dumped. Bineries where we have our models."}, {"start": 154.4, "end": 158.48000000000002, "text": " And this is just basic error checking here. Basically checking if"}, {"start": 158.48, "end": 162.16, "text": " we only have PyTorch models inside of this"}, {"start": 162.16, "end": 168.95999999999998, "text": " binaries directory. And I just create the output directory"}, {"start": 168.95999999999998, "end": 172.79999999999998, "text": " where we'll dump images. And here are the arguments. These are"}, {"start": 172.79999999999998, "end": 176.72, "text": " just the content image that you want to use, that you want to pass through the"}, {"start": 176.72, "end": 180.32, "text": " model. Then we have the, you want to set the"}, {"start": 180.32, "end": 184.88, "text": " width of the output image and it's set to 500 by default here."}, {"start": 184.88, "end": 189.35999999999999, "text": " We have default model that's called Mosaic."}, {"start": 189.35999999999999, "end": 193.6, "text": " And we just wrap all of those parameters and we call the"}, {"start": 193.6, "end": 197.44, "text": " Stylized Static Image function. Let's see what this function actually does."}, {"start": 197.44, "end": 201.12, "text": " So if we go upside here we can first see,"}, {"start": 201.12, "end": 204.64, "text": " we first want to check out if the device has a GPU"}, {"start": 204.64, "end": 211.68, "text": " or only CPU support. And then we basically just create this"}, {"start": 211.68, "end": 215.6, "text": " image path. We pass it through this function called"}, {"start": 215.6, "end": 220.88, "text": " prepareImage which will kind of add a batch dimension and do some"}, {"start": 220.88, "end": 224.4, "text": " normalization. We'll check the function a bit later."}, {"start": 224.4, "end": 228.08, "text": " And I just instantiate here the"}, {"start": 228.08, "end": 233.04000000000002, "text": " Transform model which is the actual model that performs the stylization."}, {"start": 233.04000000000002, "end": 240.0, "text": " We figure out the state. So we just load the state from a"}, {"start": 240.0, "end": 247.2, "text": " PyTorch model that's in binaries here. We print some metadata and we finally"}, {"start": 247.2, "end": 251.2, "text": " load the weights inside of the the model and we just put it into this"}, {"start": 251.2, "end": 254.64, "text": " evolve mode which is really important if you want to do inference."}, {"start": 254.64, "end": 258.32, "text": " You want to do this, you want to call this function here. And finally we just"}, {"start": 258.32, "end": 261.92, "text": " wrap this into this context torch no grad"}, {"start": 261.92, "end": 268.56, "text": " which will basically forbid PyTorch to calculate gradients which will, which"}, {"start": 268.56, "end": 273.6, "text": " would kind of be a huge memory overhead. So we call the model on the"}, {"start": 273.6, "end": 278.32, "text": " content image. We just paste it to CPU. We convert it"}, {"start": 278.32, "end": 281.36, "text": " to NumPy here. And because we had the batch dimension"}, {"start": 281.36, "end": 286.96, "text": " that we previously added we got to extract the 0th batch here."}, {"start": 286.96, "end": 290.72, "text": " And that's the stylized image and we just save the image. That's the image we"}, {"start": 290.72, "end": 294.8, "text": " saw a couple of minutes ago that was dumped here into output"}, {"start": 294.8, "end": 298.96000000000004, "text": " images folder. This one. Let's just check out that prepare image we saw in the"}, {"start": 298.96000000000004, "end": 302.40000000000003, "text": " utils here. So this is how it looks like. It basically"}, {"start": 302.40000000000003, "end": 305.2, "text": " just loads the image from the specified path"}, {"start": 305.2, "end": 309.2, "text": " specified by this variable image path. And that will just basically create a"}, {"start": 309.2, "end": 313.68, "text": " NumPy image normalized to 0 to 1 range."}, {"start": 313.68, "end": 320.16, "text": " And then we'll pretty much add the this normalization transform"}, {"start": 320.16, "end": 323.76, "text": " that PyTorch provides us with. And we'll use"}, {"start": 323.76, "end": 327.12, "text": " ImageNet's mean and standard deviation to"}, {"start": 327.12, "end": 331.92, "text": " normalize our images. After we apply the transformation here"}, {"start": 331.92, "end": 339.03999999999996, "text": " we just push the push this image onto the GPU if we have one."}, {"start": 339.03999999999996, "end": 342.08, "text": " And we just add this batch dimension in front"}, {"start": 342.08, "end": 346.24, "text": " because that's because models are always expecting"}, {"start": 346.24, "end": 349.92, "text": " this batch dimension. So pretty simple stuff."}, {"start": 349.92, "end": 354.24, "text": " In the next video I'll cover training that will be a bit more"}, {"start": 354.24, "end": 357.6, "text": " challenging. We're going to figure out how to use TensorBoard and just"}, {"start": 357.6, "end": 362.40000000000003, "text": " debug and visualize our training metrics. So let me just go"}, {"start": 362.40000000000003, "end": 366.48, "text": " ahead and try one more image here. So I'm going to use"}, {"start": 366.48, "end": 371.04, "text": " an image called figures and let me set the"}, {"start": 371.04, "end": 377.04, "text": " width here to 350 say. We'll use the same model mosaic. And if I start that one, if"}, {"start": 377.04, "end": 381.6, "text": " I run it, this is what we get. So how good the"}, {"start": 381.6, "end": 384.88, "text": " output image will be actually depends also on the size that you put here in"}, {"start": 384.88, "end": 391.28000000000003, "text": " the width. So if I put 550 say here and I run that one"}, {"start": 391.28000000000003, "end": 396.72, "text": " it will have different stylization. And it's much nicer if you ask me."}, {"start": 396.72, "end": 400.48, "text": " In the next video we'll cover training until"}, {"start": 400.48, "end": 403.68, "text": " that time just go ahead and play with this repo."}, {"start": 403.68, "end": 407.28000000000003, "text": " Try and figure out how the things work and that will be a really nice"}, {"start": 407.28000000000003, "end": 411.12, "text": " learning experience. If you like the content go ahead and subscribe,"}, {"start": 411.12, "end": 415.12, "text": " like and share the videos if you think they"}, {"start": 415.12, "end": 434.08, "text": " can bring value to somebody else. See you next time!"}]