Optimization method | Neural Style Transfer #3
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The third video in the neural style transfer series! 🎨

You'll learn about:
✔️ The optimization-based (original Gatys et al.) NST method.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ GitHub code: https://github.com/gordicaleksa/pytorch-neural-style-transfer

Relevant reading ►

(original NST paper, arxiv, old) https://arxiv.org/pdf/1508.06576.pdf
(original NST paper, CVPR, new) https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
00:12 - 02:55 Going through the readme file
02:55 - 04:30 Setting up conda environment
04:30 - 09:23 Reconstruction script (content reconstruction)
09:23 - 10:25 L-BFGS VRAM consumption consideration
10:25 - 12:37 Reconstruction script (style reconstruction)
12:44 - 18:10 Main NST script
18:10 - 19:20 Further experimenting
19:20 Outro (constructive feedback is welcome!)
'
[Credits] Music:
https://www.youtube.com/watch?v=J2X5mJ3HDYE [NCS]

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition,
rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neuralstyletransfer #deeplearning #ai
 What's up folks, we're gonna dig into some code for neural style transfer on static images using the optimization method and let's just start. Awesome, let's jump to GitHub here. You can see the URL and I've put the link in the description. So I basically wrote this repo only for this video, but it was totally worth it. I learned a lot and I hope you will benefit from it also. So it's written in PyTorch, I already mentioned that in previous video and I think it's pretty easy to use. I'm just going to briefly run through this readme file with you and in the next section you can just see the, on the left side is the output from the algorithm, on the right side are content and style inputs. And why yet another NST repo? Basically I couldn't find a reference implementation in PyTorch and other ones were really too complicated and I think this thing is really simple. And we got some examples here, just some cherry pick ones which I really liked. I think they're really neat and then following up, same thing, just in the left column is the output and on the right side is the style image that produced that output. And the following two sections are really important, they just show you how you can manipulate the weights, the respective weights. Here the first one is the style weight where you freeze the content weight and just going from left to right you can see that the amount of style in the output image is increasing. And then this is something that's rarely explained, it's a total variation loss which helps you just smoothen out the image. And then going from left to right here you can see that the image becomes really smooth. Here we can see that the way you initialize the input image, whether you use the noise, whether you use content as the initial image or style, you get different results. So on the left side you can see what you get when you start from the content image and that's usually the best way to go here. And in the middle is the random initialization and on the right side you can see the style initialization where the content from the style image actually leaked into the output image which is probably undesirable, depends if you get a really cool image that's cool. And here is the reconstruction from the original paper. I really encourage you to just go through this reading, I think it's really digestible, like it's really visual, you'll understand stuff. But I want to jump to code as soon as possible here. This is something I already explained in the last video, how we can reconstruct only content or only the style image. And it also looks really nice. And finally the setup. So this part should really be a piece of cake. You basically have only two instructions to run here. In an ideal case, in a non-ideal case you'll have to install system-wide CUDA and also minikonda which I'll be using throughout the video. Okay, let me just move the browser here and open anaconda and I'll just navigate to the place where I want to clone the repo. That's here. And I'll just do git clone from this URL here. And that should just download the repo directly. And once it downloads you can just verify it's there by typing in start like this. You can see it's here. And now we just have to navigate directly into it. Let's name it PyTorch. Yep. And we have to run only one simple command. That's conda and create. And it just went and installed conda environment for us. And now we just have to do activate PyTorch. Sorry, PyTorch.nst. Like this. And we're ready to jump into code. Yeah. Okay, so I lied. There is one more thing you need to do and that's open up your favorite IDE minus PyCharm and just connect it to the interpreter from the freshly created conda environment. And you do it like this in PyCharm. You just open up settings here and project interpreter and you just set it to PyTorch.nst. That's it. Let's go. I jumped to main function here and these are some of the default parameters you really don't need to change. This one, for example, is the default location for content images. This one is a default location for style images and I'm saving all of my images as JPEG because it saves lots of bandwidth. And for this type of output images, it also keeps the high quality of those images. Okay, so coming next are some of the parameters you'll be changing and playing with. So basically what you have here is default content image. That's this one for now. Line JPEG. And we'll be using this one for as a style. The height is fixed. 500 pixels is totally fine for this demo. Minus one means only final. So I'll put minus one here. One means save every single intermediate image. I'm using VGG19 and we'll be using LBFGS. I'll be talking more about the optimizer a bit later. Okay, so we have skipped these two here. Should reconstruct content will basically let you choose between reconstructing either content or style image. And I'll set it to true. So we'll be first reconstructing content and then the should visualize representation basically plots either feature maps or Graham matrices depending whether you picked content or style. So I'll put it to true. We want to visualize feature maps here. And so following up just we just wrap all of the all of the data in this dictionary object and we just call the function reconstruct image from representation. There's one more thing we need to do here and that's set a different content layer in VGG net. So I'll go to VGG net file here. So I'm using VGG19 right. So okay. And I'll just set four. I'll say to let's say one here. And that means we'll be using a relative to one layer. So with that being said let me just go and go ahead and run this. And this is what we get. That's a feature map. That's a first feature map from this layer. Reli to one for the line image. And let me show you the line image here from the default content directory. So this is what we are reconstructing. Let me close it and show you a couple of other feature maps here. And the reason I've chosen the relative to one is because it extracts a lower level features whereas higher higher layers would extract like higher higher level features from from images. And so you can see how it looks like. And let's see that in action. So if I close this. Oops. If I stop the program here I go to VGG and we we pick some other layer like let's say we pick com for two. Actually we'll pick yeah come for two. And if I go and start a program now this is what we get. And we can see that the feature maps are much more abstract. They tend to focus on like semantically meaningful parts of the image like in the case of Lyon it will put focus on ice as you can see here or mean or yeah like I'll just let you have a look here. And you can see like the nostrils and the eyebrows the lion has an eyebrow. My God. OK now let me stop this and change a couple of settings here. First we want to set this to one. We want to save every single image. Let me change the the layer to really to one so that we get better reconstruction. I mean less abstract reconstruction. And finally let's toggle the visualization off and let me start this. Now we'll just reconstruct the content image. So I went ahead and done that. And this is what we get. If you go here you can see the relative path here. Output images. We can see that the noise is slowly morphing into this line image as we go down the pipeline. And I went ahead and just created a video out of these images. And this is what we get here. So let me just return to the beginning because it's really fast and you can see the morphing happens really fast with LBFGS. OK just an important note on performance. So if you're using LBFGS there is a huge chance that you'll run out of video memory. So I'm using I can show you the graph here. So I'm using RTX 2080 and I guess not a lot of you folks will have a GPU that strong. So this GPU has 8 gigs of RAM and you can see here the algorithms in this repo with this configuration will eat up around 3 gigs and that's a lot. There's basically two things you can do. One is either a switch to Atom optimizer and the second one if you want to keep LBFGS because it's really good and performant with this task. You can play with resolution. So now it's I think like 500 pixels put it down to 250 300 or something. You can play with the LBFGS class itself like change the history size is 100 currently. And you can switch to VGG 16 because it will eat up less video memory. It's a smaller and shallower model than VGG 19. So now let's switch to style. Let's change a couple of parameters here. We want to visualize grand matrices and we want to pick style here. So if I run it we get the grand matrix for this image. Let me show you this one and let me close it here. And this grand matrix comes from layer RLD11 and it's just a part of the complete style representation for this image. We have five grand matrices in total which compose the general style representation. You can also see a strong line going through the main diagonal and the reason for that is because feature map. So when we take a dot product between the feature map and itself you get a high output and that's what the grand matrix actually represents. It's just like a set of different dot products between different feature maps. Let's just go ahead and see a couple of these. So this is how it looks for the next layer. It's already getting weaker in intensity because they are being normalized and the more elements they have the weaker the intensity. RLD31 and RLD41 basically won't see anything special here. Now if I exit here and once more we'll just go into reconstructing the style. Let's see what the output image looks like. So if I open it up here you can see the relative path again. I'll put images and starting from beginning it's a noise image as with the same thing as with content reconstruction. And as we go down the pipeline it gets increasingly stylized and the final image looks like this. So I went ahead and created a video and it looks like this. I encourage you to go and play ahead with this. By the way I do have a video creating function included in the repo. You just go here and you just uncomment this function. So that's it reconstructing the content and style images. Now let's jump into the neural style transfer script. Let's go. So just go ahead and open this neural style transfer file. And you can see that the script shares a lot of parameters with the reconstruction script. Basically the only three new ones and important ones are content weight, style weight and total variation weight. So you'll basically by tuning those three you'll get any image pair to like just merge together really nicely. When I say image pair I mean content style images. Let's go ahead and try it out. So I'll take a content image. I'll take the following content image. This one is called figures and I'll take the following style image. It's famous staring night from from from Van Gogh and I'll just combine those two here. So I'll put it figures JPEG and I'll put here Van Gogh's theory night JPEG and everything else here looks fine. And let me just run it. Once that's finished we get this as the output. We go to combine figures. So the name is always like this like the two images you used. That's the name of the directory where the output images will go to. And this is the result. And it looks really nice. And I'm not sure if it's pronounced. I think it's the right name to pronounce this guy's name. Name is actually Van Gogh in Dutch. Just a fun fact. Okay enough with fun facts. Let's go and see how the actual implementation looks like. So I'll go to the beginning of the function here and it starts here. So the first important bit here is the on the line 59. It's the prepare image function. So if I found find a function here you just what it does is it loads the image as an umpire rate. And then we do this transform here. Basically it's really important that you scale the image pixels with 255 because the VGG net. I'll learn during training to deal with data like that. And also normalization step is important. I just use the ones that were used for the VGG training also. So we applied the transform. We put a device to CUDA. So we copied the tensor to GPU. And then we do unsqueeze which basically just adds a dummy dimension so that it looks like a batch. And we do that for both the content and the style image. Let me return back here. So that was that part. Second we I always like the content image is the best way to initialize the image. And I just read that initial image in this variable from like a torch class. And this bit is important requires red to means that actually the this image is what is trainable. So usually when you do a machine learning training what you do is you train you tune the weights of the model here. The model weights are actually freeze. And the only thing that actually changes is the image itself. Next up we prepare the model here and we take the indices from the corresponding layers that we're using for content and star representation. So this this variable here contains both the index and the name. So say one and really one one stuff like that. And then we feed through the content and style images which were prepared previously. And finally we obtained the representation. I talked about this in the last video. And the final bit is here. So I'll just keep Adam. I'll pick one of the numerical optimizers. I'll use LBFGS here. And important. So there's just a bunch of boilerplate code here. The closure. That's something that by torch just like requires you to define. What is important here is the generation of the last build loss function. And then we just do the backward back prop on the classical like back prop on the last function that was defined. Let's take a quick look at the build loss function. So go to the implementation there. So what we do here is we take the optimization image which was initialized with the content image if you remember. And we just feed it through the VGG. We get and we get the current set of feature maps. And by just doing MSC loss between the target content representation and the current content representation we get that portion of the loss. We do a similar thing for style loss. But here we just form gray matrices out of those feature maps first and then do the MSC loss here. I just encourage you to go at your own pace through this code and understand what's happening here. So in the total loss finally is just a weighted sum of the content loss of the style loss and of the total variation loss. And that was it. So just wrapping it all up. So what we do is we have a VGG net which is frozen. We have the image which we are tuning so that the feature map is producing are getting more similar to the reference feature maps which are given by the content image and by the style image. That's it. And finally there are two things that I'd like you to go and experiment with. The first one being that you should experiment with content style and total variation losses. So I've actually put it down on a table for some like reasonable values you should use depending on the optimizer you're using. So you can see here LBFGS content in it. Just use those weights as a starting point and just go experiment. Just tweak them, increase them, decrease them and see how they affect the end result. I think that would be a really good learning experience. And the second thing would be to go to the architecture itself. So go to VGG 19 and try and experiment with different sets of layers which are using for the style representation. Try and add like a comp for three. Add more layers. Subtract some layers, whatever. Just experiment there and try and see how different sets affect the image quality. And if you find some superior representation let me know in the comments that would be really cool. So that was it for this video folks. I just encourage you to go and play with the code. That's the best way you can learn how to do this actually and understand it thoroughly. So I just encourage you also to give me like constructive feedback down in the comments. That would mean a lot to me because I'm still learning. I'm learning with you folks here. And you can tell me if the video was too long or too short which I doubt. But yeah, anyways, any comment is really welcome. Thanks a bunch. Subscribe and see you in the next video.
[{"start": 0.0, "end": 11.0, "text": " What's up folks, we're gonna dig into some code for neural style transfer on static images using the optimization method and let's just start."}, {"start": 12.0, "end": 18.0, "text": " Awesome, let's jump to GitHub here. You can see the URL and I've put the link in the description."}, {"start": 19.0, "end": 27.0, "text": " So I basically wrote this repo only for this video, but it was totally worth it. I learned a lot and I hope you will benefit from it also."}, {"start": 27.0, "end": 34.0, "text": " So it's written in PyTorch, I already mentioned that in previous video and I think it's pretty easy to use."}, {"start": 35.0, "end": 48.0, "text": " I'm just going to briefly run through this readme file with you and in the next section you can just see the, on the left side is the output from the algorithm, on the right side are content and style inputs."}, {"start": 48.0, "end": 61.0, "text": " And why yet another NST repo? Basically I couldn't find a reference implementation in PyTorch and other ones were really too complicated and I think this thing is really simple."}, {"start": 61.0, "end": 78.0, "text": " And we got some examples here, just some cherry pick ones which I really liked. I think they're really neat and then following up, same thing, just in the left column is the output and on the right side is the style image that produced that output."}, {"start": 79.0, "end": 85.0, "text": " And the following two sections are really important, they just show you how you can manipulate the weights, the respective weights."}, {"start": 85.0, "end": 97.0, "text": " Here the first one is the style weight where you freeze the content weight and just going from left to right you can see that the amount of style in the output image is increasing."}, {"start": 98.0, "end": 108.0, "text": " And then this is something that's rarely explained, it's a total variation loss which helps you just smoothen out the image."}, {"start": 108.0, "end": 127.0, "text": " And then going from left to right here you can see that the image becomes really smooth. Here we can see that the way you initialize the input image, whether you use the noise, whether you use content as the initial image or style, you get different results."}, {"start": 127.0, "end": 151.0, "text": " So on the left side you can see what you get when you start from the content image and that's usually the best way to go here. And in the middle is the random initialization and on the right side you can see the style initialization where the content from the style image actually leaked into the output image which is probably undesirable, depends if you get a really cool image that's cool."}, {"start": 151.0, "end": 165.0, "text": " And here is the reconstruction from the original paper. I really encourage you to just go through this reading, I think it's really digestible, like it's really visual, you'll understand stuff. But I want to jump to code as soon as possible here."}, {"start": 166.0, "end": 174.0, "text": " This is something I already explained in the last video, how we can reconstruct only content or only the style image. And it also looks really nice."}, {"start": 174.0, "end": 191.0, "text": " And finally the setup. So this part should really be a piece of cake. You basically have only two instructions to run here. In an ideal case, in a non-ideal case you'll have to install system-wide CUDA and also minikonda which I'll be using throughout the video."}, {"start": 191.0, "end": 212.0, "text": " Okay, let me just move the browser here and open anaconda and I'll just navigate to the place where I want to clone the repo. That's here. And I'll just do git clone from this URL here. And that should just download the repo directly."}, {"start": 212.0, "end": 222.0, "text": " And once it downloads you can just verify it's there by typing in start like this. You can see it's here. And now we just have to navigate directly into it."}, {"start": 222.0, "end": 243.0, "text": " Let's name it PyTorch. Yep. And we have to run only one simple command. That's conda and create. And it just went and installed conda environment for us. And now we just have to do activate PyTorch. Sorry, PyTorch.nst."}, {"start": 243.0, "end": 261.0, "text": " Like this. And we're ready to jump into code. Yeah. Okay, so I lied. There is one more thing you need to do and that's open up your favorite IDE minus PyCharm and just connect it to the interpreter from the freshly created conda environment."}, {"start": 262.0, "end": 271.0, "text": " And you do it like this in PyCharm. You just open up settings here and project interpreter and you just set it to PyTorch.nst. That's it. Let's go."}, {"start": 271.0, "end": 283.0, "text": " I jumped to main function here and these are some of the default parameters you really don't need to change. This one, for example, is the default location for content images."}, {"start": 284.0, "end": 297.0, "text": " This one is a default location for style images and I'm saving all of my images as JPEG because it saves lots of bandwidth. And for this type of output images, it also keeps the high quality of those images."}, {"start": 297.0, "end": 310.0, "text": " Okay, so coming next are some of the parameters you'll be changing and playing with. So basically what you have here is default content image. That's this one for now. Line JPEG."}, {"start": 310.0, "end": 329.0, "text": " And we'll be using this one for as a style. The height is fixed. 500 pixels is totally fine for this demo. Minus one means only final. So I'll put minus one here. One means save every single intermediate image."}, {"start": 329.0, "end": 349.0, "text": " I'm using VGG19 and we'll be using LBFGS. I'll be talking more about the optimizer a bit later. Okay, so we have skipped these two here. Should reconstruct content will basically let you choose between reconstructing either content or style image."}, {"start": 349.0, "end": 362.0, "text": " And I'll set it to true. So we'll be first reconstructing content and then the should visualize representation basically plots either feature maps or Graham matrices depending whether you picked content or style."}, {"start": 363.0, "end": 376.0, "text": " So I'll put it to true. We want to visualize feature maps here. And so following up just we just wrap all of the all of the data in this dictionary object and we just call the function reconstruct image from representation."}, {"start": 376.0, "end": 396.0, "text": " There's one more thing we need to do here and that's set a different content layer in VGG net. So I'll go to VGG net file here. So I'm using VGG19 right. So okay. And I'll just set four. I'll say to let's say one here."}, {"start": 396.0, "end": 412.0, "text": " And that means we'll be using a relative to one layer. So with that being said let me just go and go ahead and run this. And this is what we get. That's a feature map. That's a first feature map from this layer."}, {"start": 412.0, "end": 427.0, "text": " Reli to one for the line image. And let me show you the line image here from the default content directory. So this is what we are reconstructing. Let me close it and show you a couple of other feature maps here."}, {"start": 427.0, "end": 443.0, "text": " And the reason I've chosen the relative to one is because it extracts a lower level features whereas higher higher layers would extract like higher higher level features from from images."}, {"start": 443.0, "end": 458.0, "text": " And so you can see how it looks like. And let's see that in action. So if I close this. Oops. If I stop the program here I go to VGG and we we pick some other layer like let's say we pick com for two."}, {"start": 458.0, "end": 473.0, "text": " Actually we'll pick yeah come for two. And if I go and start a program now this is what we get. And we can see that the feature maps are much more abstract."}, {"start": 473.0, "end": 493.0, "text": " They tend to focus on like semantically meaningful parts of the image like in the case of Lyon it will put focus on ice as you can see here or mean or yeah like I'll just let you have a look here."}, {"start": 493.0, "end": 506.0, "text": " And you can see like the nostrils and the eyebrows the lion has an eyebrow. My God. OK now let me stop this and change a couple of settings here."}, {"start": 507.0, "end": 520.0, "text": " First we want to set this to one. We want to save every single image. Let me change the the layer to really to one so that we get better reconstruction."}, {"start": 520.0, "end": 531.0, "text": " I mean less abstract reconstruction. And finally let's toggle the visualization off and let me start this. Now we'll just reconstruct the content image."}, {"start": 532.0, "end": 539.0, "text": " So I went ahead and done that. And this is what we get. If you go here you can see the relative path here. Output images."}, {"start": 539.0, "end": 552.0, "text": " We can see that the noise is slowly morphing into this line image as we go down the pipeline. And I went ahead and just created a video out of these images."}, {"start": 553.0, "end": 562.0, "text": " And this is what we get here. So let me just return to the beginning because it's really fast and you can see the morphing happens really fast with LBFGS."}, {"start": 562.0, "end": 569.0, "text": " OK just an important note on performance. So if you're using LBFGS there is a huge chance that you'll run out of video memory."}, {"start": 570.0, "end": 578.0, "text": " So I'm using I can show you the graph here. So I'm using RTX 2080 and I guess not a lot of you folks will have a GPU that strong."}, {"start": 579.0, "end": 587.0, "text": " So this GPU has 8 gigs of RAM and you can see here the algorithms in this repo with this configuration will eat up around 3 gigs and that's a lot."}, {"start": 587.0, "end": 599.0, "text": " There's basically two things you can do. One is either a switch to Atom optimizer and the second one if you want to keep LBFGS because it's really good and performant with this task."}, {"start": 600.0, "end": 613.0, "text": " You can play with resolution. So now it's I think like 500 pixels put it down to 250 300 or something. You can play with the LBFGS class itself like change the history size is 100 currently."}, {"start": 613.0, "end": 623.0, "text": " And you can switch to VGG 16 because it will eat up less video memory. It's a smaller and shallower model than VGG 19."}, {"start": 624.0, "end": 634.0, "text": " So now let's switch to style. Let's change a couple of parameters here. We want to visualize grand matrices and we want to pick style here."}, {"start": 634.0, "end": 647.0, "text": " So if I run it we get the grand matrix for this image. Let me show you this one and let me close it here."}, {"start": 648.0, "end": 657.0, "text": " And this grand matrix comes from layer RLD11 and it's just a part of the complete style representation for this image."}, {"start": 657.0, "end": 664.0, "text": " We have five grand matrices in total which compose the general style representation."}, {"start": 665.0, "end": 671.0, "text": " You can also see a strong line going through the main diagonal and the reason for that is because feature map."}, {"start": 672.0, "end": 679.0, "text": " So when we take a dot product between the feature map and itself you get a high output and that's what the grand matrix actually represents."}, {"start": 680.0, "end": 684.0, "text": " It's just like a set of different dot products between different feature maps."}, {"start": 684.0, "end": 689.0, "text": " Let's just go ahead and see a couple of these. So this is how it looks for the next layer."}, {"start": 690.0, "end": 698.0, "text": " It's already getting weaker in intensity because they are being normalized and the more elements they have the weaker the intensity."}, {"start": 699.0, "end": 704.0, "text": " RLD31 and RLD41 basically won't see anything special here."}, {"start": 705.0, "end": 709.0, "text": " Now if I exit here and once more we'll just go into reconstructing the style."}, {"start": 709.0, "end": 717.0, "text": " Let's see what the output image looks like. So if I open it up here you can see the relative path again."}, {"start": 718.0, "end": 725.0, "text": " I'll put images and starting from beginning it's a noise image as with the same thing as with content reconstruction."}, {"start": 726.0, "end": 733.0, "text": " And as we go down the pipeline it gets increasingly stylized and the final image looks like this."}, {"start": 733.0, "end": 739.0, "text": " So I went ahead and created a video and it looks like this."}, {"start": 740.0, "end": 745.0, "text": " I encourage you to go and play ahead with this."}, {"start": 746.0, "end": 750.0, "text": " By the way I do have a video creating function included in the repo."}, {"start": 751.0, "end": 757.0, "text": " You just go here and you just uncomment this function."}, {"start": 757.0, "end": 765.0, "text": " So that's it reconstructing the content and style images. Now let's jump into the neural style transfer script. Let's go."}, {"start": 766.0, "end": 770.0, "text": " So just go ahead and open this neural style transfer file."}, {"start": 771.0, "end": 776.0, "text": " And you can see that the script shares a lot of parameters with the reconstruction script."}, {"start": 777.0, "end": 784.0, "text": " Basically the only three new ones and important ones are content weight, style weight and total variation weight."}, {"start": 784.0, "end": 791.0, "text": " So you'll basically by tuning those three you'll get any image pair to like just merge together really nicely."}, {"start": 792.0, "end": 794.0, "text": " When I say image pair I mean content style images."}, {"start": 795.0, "end": 800.0, "text": " Let's go ahead and try it out. So I'll take a content image. I'll take the following content image."}, {"start": 801.0, "end": 805.0, "text": " This one is called figures and I'll take the following style image."}, {"start": 805.0, "end": 815.0, "text": " It's famous staring night from from from Van Gogh and I'll just combine those two here."}, {"start": 816.0, "end": 832.0, "text": " So I'll put it figures JPEG and I'll put here Van Gogh's theory night JPEG and everything else here looks fine."}, {"start": 832.0, "end": 834.0, "text": " And let me just run it."}, {"start": 835.0, "end": 843.0, "text": " Once that's finished we get this as the output. We go to combine figures."}, {"start": 844.0, "end": 848.0, "text": " So the name is always like this like the two images you used."}, {"start": 849.0, "end": 852.0, "text": " That's the name of the directory where the output images will go to."}, {"start": 853.0, "end": 856.0, "text": " And this is the result. And it looks really nice."}, {"start": 856.0, "end": 862.0, "text": " And I'm not sure if it's pronounced. I think it's the right name to pronounce this guy's name."}, {"start": 863.0, "end": 866.0, "text": " Name is actually Van Gogh in Dutch. Just a fun fact."}, {"start": 867.0, "end": 871.0, "text": " Okay enough with fun facts. Let's go and see how the actual implementation looks like."}, {"start": 872.0, "end": 878.0, "text": " So I'll go to the beginning of the function here and it starts here."}, {"start": 879.0, "end": 884.0, "text": " So the first important bit here is the on the line 59. It's the prepare image function."}, {"start": 884.0, "end": 892.0, "text": " So if I found find a function here you just what it does is it loads the image as an umpire rate."}, {"start": 893.0, "end": 902.0, "text": " And then we do this transform here. Basically it's really important that you scale the image pixels with 255 because the VGG net."}, {"start": 903.0, "end": 907.0, "text": " I'll learn during training to deal with data like that. And also normalization step is important."}, {"start": 908.0, "end": 911.0, "text": " I just use the ones that were used for the VGG training also."}, {"start": 911.0, "end": 918.0, "text": " So we applied the transform. We put a device to CUDA. So we copied the tensor to GPU."}, {"start": 919.0, "end": 924.0, "text": " And then we do unsqueeze which basically just adds a dummy dimension so that it looks like a batch."}, {"start": 925.0, "end": 930.0, "text": " And we do that for both the content and the style image. Let me return back here. So that was that part."}, {"start": 931.0, "end": 936.0, "text": " Second we I always like the content image is the best way to initialize the image."}, {"start": 936.0, "end": 940.0, "text": " And I just read that initial image in this variable from like a torch class."}, {"start": 941.0, "end": 947.0, "text": " And this bit is important requires red to means that actually the this image is what is trainable."}, {"start": 948.0, "end": 954.0, "text": " So usually when you do a machine learning training what you do is you train you tune the weights of the model here."}, {"start": 955.0, "end": 961.0, "text": " The model weights are actually freeze. And the only thing that actually changes is the image itself."}, {"start": 961.0, "end": 972.0, "text": " Next up we prepare the model here and we take the indices from the corresponding layers that we're using for content and star representation."}, {"start": 973.0, "end": 979.0, "text": " So this this variable here contains both the index and the name. So say one and really one one stuff like that."}, {"start": 980.0, "end": 985.0, "text": " And then we feed through the content and style images which were prepared previously."}, {"start": 985.0, "end": 990.0, "text": " And finally we obtained the representation. I talked about this in the last video."}, {"start": 991.0, "end": 998.0, "text": " And the final bit is here. So I'll just keep Adam. I'll pick one of the numerical optimizers. I'll use LBFGS here."}, {"start": 999.0, "end": 1008.0, "text": " And important. So there's just a bunch of boilerplate code here. The closure. That's something that by torch just like requires you to define."}, {"start": 1008.0, "end": 1021.0, "text": " What is important here is the generation of the last build loss function. And then we just do the backward back prop on the classical like back prop on the last function that was defined."}, {"start": 1022.0, "end": 1032.0, "text": " Let's take a quick look at the build loss function. So go to the implementation there. So what we do here is we take the optimization image which was initialized with the content image if you remember."}, {"start": 1032.0, "end": 1037.0, "text": " And we just feed it through the VGG. We get and we get the current set of feature maps."}, {"start": 1038.0, "end": 1047.0, "text": " And by just doing MSC loss between the target content representation and the current content representation we get that portion of the loss."}, {"start": 1048.0, "end": 1058.0, "text": " We do a similar thing for style loss. But here we just form gray matrices out of those feature maps first and then do the MSC loss here."}, {"start": 1058.0, "end": 1063.0, "text": " I just encourage you to go at your own pace through this code and understand what's happening here."}, {"start": 1064.0, "end": 1071.0, "text": " So in the total loss finally is just a weighted sum of the content loss of the style loss and of the total variation loss."}, {"start": 1072.0, "end": 1078.0, "text": " And that was it. So just wrapping it all up. So what we do is we have a VGG net which is frozen."}, {"start": 1078.0, "end": 1090.0, "text": " We have the image which we are tuning so that the feature map is producing are getting more similar to the reference feature maps which are given by the content image and by the style image."}, {"start": 1091.0, "end": 1095.0, "text": " That's it. And finally there are two things that I'd like you to go and experiment with."}, {"start": 1096.0, "end": 1101.0, "text": " The first one being that you should experiment with content style and total variation losses."}, {"start": 1101.0, "end": 1110.0, "text": " So I've actually put it down on a table for some like reasonable values you should use depending on the optimizer you're using."}, {"start": 1111.0, "end": 1119.0, "text": " So you can see here LBFGS content in it. Just use those weights as a starting point and just go experiment."}, {"start": 1120.0, "end": 1125.0, "text": " Just tweak them, increase them, decrease them and see how they affect the end result."}, {"start": 1126.0, "end": 1129.0, "text": " I think that would be a really good learning experience."}, {"start": 1129.0, "end": 1133.0, "text": " And the second thing would be to go to the architecture itself."}, {"start": 1134.0, "end": 1142.0, "text": " So go to VGG 19 and try and experiment with different sets of layers which are using for the style representation."}, {"start": 1143.0, "end": 1149.0, "text": " Try and add like a comp for three. Add more layers. Subtract some layers, whatever."}, {"start": 1150.0, "end": 1153.0, "text": " Just experiment there and try and see how different sets affect the image quality."}, {"start": 1153.0, "end": 1158.0, "text": " And if you find some superior representation let me know in the comments that would be really cool."}, {"start": 1159.0, "end": 1164.0, "text": " So that was it for this video folks. I just encourage you to go and play with the code."}, {"start": 1165.0, "end": 1170.0, "text": " That's the best way you can learn how to do this actually and understand it thoroughly."}, {"start": 1171.0, "end": 1175.0, "text": " So I just encourage you also to give me like constructive feedback down in the comments."}, {"start": 1176.0, "end": 1180.0, "text": " That would mean a lot to me because I'm still learning. I'm learning with you folks here."}, {"start": 1180.0, "end": 1185.0, "text": " And you can tell me if the video was too long or too short which I doubt."}, {"start": 1186.0, "end": 1189.0, "text": " But yeah, anyways, any comment is really welcome."}, {"start": 1189.0, "end": 1210.0, "text": " Thanks a bunch. Subscribe and see you in the next video."}]