Basic Theory | Neural Style Transfer #2
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The second video in the neural style transfer series! 🎨

You'll learn about:
✔️ The basic theory behind how neural style transfer works 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
I hope the video provides you with a strong basic intuition and understanding, but for those of you who want to take it further here are some additional materials relevant to this video:

papers ►
✔️(original NST paper, arxiv, old) https://arxiv.org/pdf/1508.06576.pdf
✔️(original NST paper, CVPR, new) https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf 

blogs/articles ►
✔️ History of style transfer (3 part blog series by Adobe's Aaron Hertzmann) https://research.adobe.com/news/image-stylization-history-and-future/
✔️Nice overview of fast NST algorithms https://www.fritz.ai/style-transfer/

Note: It seems that YouTube's video transcoding process messed up the intro and outro NST clips - they look much nicer and higher quality on my machine.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
00:00 - intro & NST series overview 
02:25 - what I want this series to be 
03:30 - defining the task of NST 
04:01 - 2 types of style transfer 
04:43 - a glimpse of the image style transfer history 
06:55 - explanation of the content representation 
10:10 - explanation of the style representation 
14:12 - putting it all together (animation) 

[Credits] Music:
https://www.youtube.com/watch?v=J2X5mJ3HDYE [NCS]

[Credits] Images: 
Found the useful Gram matrix intuition image in this blog: https://towardsdatascience.com/light-on-math-machine-learning-intuitive-guide-to-neural-style-transfer-ef88e46697ee 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition,
rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neuralstyletransfer #deeplearning #ai
 Welcome to the second video in this video series on neural style transfer where you're gonna learn how to do this and Let's jump into the video So this video will give you a deeper understanding of the basic Neural style transfer theory, but before I go there I'd like to give you an overview of the whole series and if you only came for this video feel free to skip directly to it so Let's start So last video was more of a teaser of showing you all the things that neural style transfer can actually do and as I already mentioned this one will be about basic theory and The third one will be about static image neural style transfer using the optimization method LBFGS or or Adam numerical optimizers, whatever so and And in general the the first part of the series will focus on static image neural style transfer whereas the second part will focus on videos and Fourth will be the second the appendix pretty much to this video on a more advanced and neural style transfer theory and Fifth one will focus on so not using the optimization method, but using CNN's You just plug in an image input as an input and you get a stylized image out and I'm also going to teach you how to train your own models so that you can use different styles Then we'll talk a little bit about segmentation which will help you stylize only certain portions of the of the image and Then we'll jump into the videos part starting with primitive video where We're going to learn how to apply it on a per frame basis without using any temporal loss But then we'll start using Including the temporal loss itself inside the models and we'll get much more stable model there and the output also of course The tenth will focus on training those models and the last one in this series will be about going deeper in general like try and use some some other family of models like mobile nets efficient nets some state-of-the-art models and see if that Gives us better results in general Next off I want to tell you more about what I want this series to actually be So I want to be code heavy. It's going to be really practical and except for this video and the advanced theory one and I want to keep it simple. I want to only use PyTorch as a framework and Python as the programming language. So no dual boots no system dependent scripts and no exotic languages such as Lua obsolete frameworks such as torture cafe and no tensorflow even though it's still relevant Especially with a 2.0 version, but I just want to pick one and I think PyTorch is being the battle and it's much more It's nicer to write in Code will be shared through my github repo So you have you can just I want to make it. I want to make it really simple. You can just get clone my repo Create the environment file Create environment using my environment file and that's it. You can start playing straight ahead So that's the end of the series overview now. Let's jump straight ahead into the video itself And let me start off with defining what the actual task will will be so we get one image as As the first input which contains the content that we want to preserve We get a second image which has the style That we want to transfer to this content image. We combine them where the plus denotes Neural style transfer transform and what we get out is composite image That's a stylized version of the content image and that's it. That's a task next off, let's see two basic types of style transfer and The first one is the one I'll be showing in this video series It's the artistic style transfer where the style where the style image we we want to use Is actually artistic image It can be either a cartoonish or some drawings and painting Whatever and the second type of style transfer is the photorealistic style transfer where both of the images are actually real and we try to just just mimic the style of the one of these Onto another and get a composite image out as you can see here on the screen so I thought it'd be worth including some history here and Basically, there's a difference between a style transfer and neural style transfer style transfer is something that's been going on for decades now already and Neural style transfer is just the same thing but using neural neural nets and it all started in natives pretty much where people were using simple signal processing techniques and filters to to get a stylized images out like the one you see here and Then in 2000s they started applying patch based methods like the one here called image analogies Where you need to have image pairs? So the content image in the stylized version and then given the new content image you can stylize it the same as the pair That was previously given and this method gave some decent results But only in 2015 that we get to the neural style transfer I applying ConvNets to do the same thing of transferring and it outperformed every other approach previously developed And now to the core NST algorithm itself. So where it all started it all started in 2015 where Leung Gaitis and his colleagues wrote this research paper titled a neural algorithm of artistic style and What the key finding of the paper was is that the content and the style representations can be decoupled inside a CNN architecture and Specifically a VGG net played a key role in this in this paper and you can see the architecture on the screen a bit more detail on the VGG network itself. So it was trained on the image net data set for the tests of image classification and object localization But it actually wasn't a winner on that year's Classification challenge. It was a first runner-up the first network to the one the competition was Google net or Google lenit But VGG did one the localization tasks Let's see what role VGG had in this NST paper So it helped create a rich and robust Representation of the semantics of the input image. So how we find the counter representation is the following We take some images and input we feed it through the CNN the VGG net here and we take the feature maps from a certain layer like like say a conf of 41 and those feature maps are what it represents the content of the input image and it's really that easy and Just for the sake of making feature maps less abstract Let's see how they actually look like for this concrete image for for this line image And you can see why they are called feature maps They can basically be interpreted as images and they contain either a low level detail such as edges Stuff like that or high level details depending from which layer of the VGG net or in general CNN Do you extract them out? Okay, and now for the fun part. So let's take a Gaussian noise image which will eventually become the stylized image that we want and Feed it through the VGG and we'll get its counter representation, which is currently rubbish Let's see how we can drive it So it has the same representation as the input image so we get those two images in Zimba We feed them through the VGG and we get their feature maps Which are as I already mentioned the content representation the current content representation of those two images We we can flatten those feature maps So each feature map becomes a row and this output matrix and now we need to drive those P and F matrix To be the same and we accomplish that using this loss, which is a simple MSE loss where you would just Take a element-wise subtraction and we do element-wise Squaring on those elements and we just try and drive that that loss to zero. I would try to minimize it And now for the really fun part And we're going to see what happens when we drive the loss down to zero And I'll just give you a couple of seconds to watch the animations So what you can see on the screen is on the left side You see what happens when the you see that the F matrix is getting closer to the P matrix Which is equivalent to the loss getting down to zero and on the right you can see the optimization procedure itself No, it's image becoming slowly becoming the input image The bottom animation is just the whole optimization procedure Whereas the upper animation is just the initial part of the optimization procedure in a slow-mo Because it happens really fast using LBFGS optimizer what you can see on the next screen is that the LBFGS is much more is much faster than the atom optimizer and in only 100 iterations LBFGS already Seems to be a morphing this noise image into content image Whereas atom is only just beginning to do that now for the second most important idea in this video And that's how do we capture the style of an image? So how do we find its style representation? So we have this style image input style image We feed it through this VGG net and we get a set of feature maps this time Taking those from starting from layer com 1 1 and going through layer com 5 1 And what we do is we construct this feature space over these feature maps using something called gram transform So we create gram matrices out of those feature maps and The set of those gram matrices is what ultimately represents the style of the image Or let's call it the style representation and now you might ask what's a what's a gram matrix and that's a legit question So I tick a style image as an input I fed it through the VGG and from one of those layers I constructed a gram matrix, and this is how it looks like this is exactly how it looks like and It answers an important question and that's which feature maps are used in the VGG And it answers an important question and that's which feature maps tend to activate together we already saw how the feature maps look like and In one of the previous slides and now we have an answer to this question so it's a simple covariance matrix between different feature maps and the way we calculate an element in this matrix is by just doing a dot product between two feature maps and That just captures the the texture information as it turns out Let me give you some more intuition behind why gram matrix actually works So here's a hypothetical example where on the upper row we have a Hypothetical output three feature maps and on the bottom row the same thing or just for some other input image And if we would take element wise subtraction between those two rows would get a nonzero output which means we have a nonzero content loss Which means that the input images have different semantics right different content So the dog is upside down on the bottom row But on the other hand if you take a look on the right side, you'll see that the gram matrices are actually the same Which means that the two input images are stylized in the same manner, which is kind of true if you take a look at it and The style loss will be zero because of that So let's make it more explicit how we calculate the style loss So we have the input style image. We have the input noise image We feed them through a VGG. We'll get a set of feature maps here I'm only showing for simplicity just one set of feature maps we construct gram matrices over those feature maps And what we do is just a simple MSC loss again Which is just a element wise projection followed by element wise squared followed by element wise squaring and the final Style loss is actually just a weighted sum of those Terms for every layer in the network and that's it now. I'll do the same thing as a different condom Which let's see what happens when the style representation of the input noise image becomes the same as the style Representation of the input style image, and I'll give you a couple of seconds to just watch the animations watch closely And you can see there's a spike there so What happens is on the left side? G represents the set of gram matrices of the input noise image and a represents the set of Gram matrices of the input style image and as they are getting closer to each other On the right side you can see an animation where an input noise image Initially input noise image is slowly becoming Stylized it's capturing the style of this simple style image although. It's district disregarding the Semantics is just capturing the style. That's it and now putting it all together So the total loss is a way a combination of a common loss and the style loss And what it basically says is the following we want the input noise image you have the same style representation as the input style image and to have the same content representation as the input condom image and That objective might not be fully minimizable because a there does not exist a solution or b We cannot find a solution But still we'll get a visual appearance that we want and just take a look at the animation here And the line is slowly appearing In that style image and we are getting the composite image out that we and that was that was like the whole point of this Video so that's it for the second video if you like my content consider subscribing Gently push that like button and see you in the next video
[{"start": 0.0, "end": 6.4, "text": " Welcome to the second video in this video series on neural style transfer where you're gonna learn how to do this and"}, {"start": 7.04, "end": 9.040000000000001, "text": " Let's jump into the video"}, {"start": 9.68, "end": 13.780000000000001, "text": " So this video will give you a deeper understanding of the basic"}, {"start": 14.44, "end": 17.32, "text": " Neural style transfer theory, but before I go there"}, {"start": 17.32, "end": 23.48, "text": " I'd like to give you an overview of the whole series and if you only came for this video feel free to skip"}, {"start": 23.96, "end": 25.44, "text": " directly to it"}, {"start": 25.44, "end": 27.080000000000002, "text": " so"}, {"start": 27.080000000000002, "end": 28.52, "text": " Let's start"}, {"start": 28.52, "end": 35.16, "text": " So last video was more of a teaser of showing you all the things that neural style transfer can actually do and"}, {"start": 35.64, "end": 39.92, "text": " as I already mentioned this one will be about basic theory and"}, {"start": 40.8, "end": 43.879999999999995, "text": " The third one will be about static image neural style transfer"}, {"start": 44.519999999999996, "end": 46.519999999999996, "text": " using the optimization method"}, {"start": 47.519999999999996, "end": 52.44, "text": " LBFGS or or Adam numerical optimizers, whatever so and"}, {"start": 52.44, "end": 58.16, "text": " And in general the the first part of the series will focus on static image neural style transfer"}, {"start": 58.4, "end": 61.4, "text": " whereas the second part will focus on videos and"}, {"start": 63.12, "end": 70.28, "text": " Fourth will be the second the appendix pretty much to this video on a more advanced and neural style transfer theory and"}, {"start": 71.36, "end": 76.9, "text": " Fifth one will focus on so not using the optimization method, but using CNN's"}, {"start": 76.9, "end": 82.54, "text": " You just plug in an image input as an input and you get a stylized image out and"}, {"start": 83.34, "end": 87.68, "text": " I'm also going to teach you how to train your own models so that you can"}, {"start": 88.34, "end": 90.34, "text": " use different styles"}, {"start": 91.02000000000001, "end": 93.78, "text": " Then we'll talk a little bit about segmentation"}, {"start": 94.66000000000001, "end": 101.06, "text": " which will help you stylize only certain portions of the of the image and"}, {"start": 101.06, "end": 106.66, "text": " Then we'll jump into the videos part starting with primitive video where"}, {"start": 107.58, "end": 112.02000000000001, "text": " We're going to learn how to apply it on a per frame basis without using any temporal loss"}, {"start": 112.38, "end": 114.62, "text": " But then we'll start using"}, {"start": 115.18, "end": 120.86, "text": " Including the temporal loss itself inside the models and we'll get much more stable model there"}, {"start": 121.66, "end": 123.02000000000001, "text": " and"}, {"start": 123.02000000000001, "end": 125.02000000000001, "text": " the output also of course"}, {"start": 125.02, "end": 132.57999999999998, "text": " The tenth will focus on training those models and the last one in this series will be about going deeper in general"}, {"start": 132.57999999999998, "end": 140.74, "text": " like try and use some some other family of models like mobile nets efficient nets some state-of-the-art models and see if that"}, {"start": 141.3, "end": 143.96, "text": " Gives us better results in general"}, {"start": 145.26, "end": 150.04, "text": " Next off I want to tell you more about what I want this series to actually be"}, {"start": 150.04, "end": 154.72, "text": " So I want to be code heavy. It's going to be really practical and"}, {"start": 155.67999999999998, "end": 158.68, "text": " except for this video and the advanced theory one and"}, {"start": 159.92, "end": 161.92, "text": " I want to keep it simple. I want to only use"}, {"start": 162.76, "end": 167.72, "text": " PyTorch as a framework and Python as the programming language. So no dual boots no"}, {"start": 168.35999999999999, "end": 171.39999999999998, "text": " system dependent scripts and no exotic languages such as Lua"}, {"start": 171.92, "end": 177.35999999999999, "text": " obsolete frameworks such as torture cafe and no tensorflow even though it's still relevant"}, {"start": 177.36, "end": 184.8, "text": " Especially with a 2.0 version, but I just want to pick one and I think PyTorch is being the battle and it's much more"}, {"start": 184.92000000000002, "end": 186.92000000000002, "text": " It's nicer to write in"}, {"start": 187.44000000000003, "end": 189.98000000000002, "text": " Code will be shared through my github repo"}, {"start": 190.36, "end": 197.0, "text": " So you have you can just I want to make it. I want to make it really simple. You can just get clone my repo"}, {"start": 197.52, "end": 199.52, "text": " Create the environment file"}, {"start": 199.68, "end": 204.20000000000002, "text": " Create environment using my environment file and that's it. You can start playing straight ahead"}, {"start": 204.2, "end": 209.48, "text": " So that's the end of the series overview now. Let's jump straight ahead into the video itself"}, {"start": 209.48, "end": 216.16, "text": " And let me start off with defining what the actual task will will be so we get one image as"}, {"start": 216.32, "end": 220.56, "text": " As the first input which contains the content that we want to preserve"}, {"start": 221.12, "end": 223.76, "text": " We get a second image which has the style"}, {"start": 224.28, "end": 229.64, "text": " That we want to transfer to this content image. We combine them where the plus denotes"}, {"start": 229.64, "end": 234.11999999999998, "text": " Neural style transfer transform and what we get out is"}, {"start": 235.0, "end": 236.27999999999997, "text": " composite image"}, {"start": 236.27999999999997, "end": 240.48, "text": " That's a stylized version of the content image and that's it. That's a task"}, {"start": 241.6, "end": 246.32, "text": " next off, let's see two basic types of style transfer and"}, {"start": 248.11999999999998, "end": 251.64, "text": " The first one is the one I'll be showing in this video series"}, {"start": 251.64, "end": 257.08, "text": " It's the artistic style transfer where the style where the style image we we want to use"}, {"start": 257.08, "end": 259.8, "text": " Is actually artistic image"}, {"start": 260.4, "end": 263.68, "text": " It can be either a cartoonish or some drawings and painting"}, {"start": 264.15999999999997, "end": 270.96, "text": " Whatever and the second type of style transfer is the photorealistic style transfer where both of the images are actually"}, {"start": 271.59999999999997, "end": 275.71999999999997, "text": " real and we try to just just mimic the style of the"}, {"start": 276.36, "end": 278.08, "text": " one of these"}, {"start": 278.08, "end": 282.47999999999996, "text": " Onto another and get a composite image out as you can see here on the screen"}, {"start": 282.48, "end": 286.96000000000004, "text": " so I thought it'd be worth including some history here and"}, {"start": 287.44, "end": 295.24, "text": " Basically, there's a difference between a style transfer and neural style transfer style transfer is something that's been going on for decades now already and"}, {"start": 295.56, "end": 304.20000000000005, "text": " Neural style transfer is just the same thing but using neural neural nets and it all started in natives pretty much where people were using"}, {"start": 304.96000000000004, "end": 306.28000000000003, "text": " simple"}, {"start": 306.28, "end": 312.79999999999995, "text": " signal processing techniques and filters to to get a stylized images out like the one you see here and"}, {"start": 313.47999999999996, "end": 319.76, "text": " Then in 2000s they started applying patch based methods like the one here called image analogies"}, {"start": 319.88, "end": 322.2, "text": " Where you need to have image pairs?"}, {"start": 322.2, "end": 329.96, "text": " So the content image in the stylized version and then given the new content image you can stylize it the same as the pair"}, {"start": 330.44, "end": 335.15999999999997, "text": " That was previously given and this method gave some decent results"}, {"start": 335.16, "end": 339.90000000000003, "text": " But only in 2015 that we get to the neural style transfer"}, {"start": 340.44, "end": 342.44, "text": " I applying"}, {"start": 342.48, "end": 349.52000000000004, "text": " ConvNets to do the same thing of transferring and it outperformed every other approach previously developed"}, {"start": 350.24, "end": 352.24, "text": " And now to the core"}, {"start": 352.32000000000005, "end": 357.04, "text": " NST algorithm itself. So where it all started it all started in"}, {"start": 358.08000000000004, "end": 359.48, "text": " 2015 where"}, {"start": 359.48, "end": 365.8, "text": " Leung Gaitis and his colleagues wrote this research paper titled a neural algorithm of artistic style and"}, {"start": 366.68, "end": 373.96000000000004, "text": " What the key finding of the paper was is that the content and the style representations can be decoupled inside a"}, {"start": 374.48, "end": 376.48, "text": " CNN architecture and"}, {"start": 376.76, "end": 383.26, "text": " Specifically a VGG net played a key role in this in this paper and you can see the architecture"}, {"start": 383.64000000000004, "end": 386.76, "text": " on the screen a bit more detail on the"}, {"start": 386.76, "end": 394.88, "text": " VGG network itself. So it was trained on the image net data set for the tests of image classification"}, {"start": 395.4, "end": 397.4, "text": " and object localization"}, {"start": 398.12, "end": 401.88, "text": " But it actually wasn't a winner on that year's"}, {"start": 403.03999999999996, "end": 410.15999999999997, "text": " Classification challenge. It was a first runner-up the first network to the one the competition was Google net or Google lenit"}, {"start": 410.68, "end": 414.36, "text": " But VGG did one the localization tasks"}, {"start": 414.36, "end": 417.92, "text": " Let's see what role VGG had in this NST paper"}, {"start": 419.96000000000004, "end": 421.96000000000004, "text": " So it helped create a"}, {"start": 422.8, "end": 424.8, "text": " rich and robust"}, {"start": 425.04, "end": 430.6, "text": " Representation of the semantics of the input image. So how we find the counter representation is the following"}, {"start": 430.6, "end": 435.0, "text": " We take some images and input we feed it through the CNN"}, {"start": 435.52000000000004, "end": 443.0, "text": " the VGG net here and we take the feature maps from a certain layer like like say a conf of 41 and"}, {"start": 443.0, "end": 449.28, "text": " those feature maps are what it represents the content of the input image and it's really that easy and"}, {"start": 449.8, "end": 452.48, "text": " Just for the sake of making feature maps less abstract"}, {"start": 452.48, "end": 456.72, "text": " Let's see how they actually look like for this concrete image for for this line image"}, {"start": 457.44, "end": 459.96, "text": " And you can see why they are called feature maps"}, {"start": 460.44, "end": 466.88, "text": " They can basically be interpreted as images and they contain either a low level detail such as edges"}, {"start": 466.88, "end": 473.68, "text": " Stuff like that or high level details depending from which layer of the VGG net or in general CNN"}, {"start": 473.68, "end": 478.2, "text": " Do you extract them out? Okay, and now for the fun part. So let's take a"}, {"start": 479.0, "end": 483.68, "text": " Gaussian noise image which will eventually become the stylized image that we want and"}, {"start": 484.48, "end": 490.0, "text": " Feed it through the VGG and we'll get its counter representation, which is currently rubbish"}, {"start": 490.36, "end": 492.08, "text": " Let's see how we can drive it"}, {"start": 492.08, "end": 498.47999999999996, "text": " So it has the same representation as the input image so we get those two images in Zimba"}, {"start": 498.96, "end": 502.84, "text": " We feed them through the VGG and we get their feature maps"}, {"start": 503.15999999999997, "end": 507.28, "text": " Which are as I already mentioned the content representation the current"}, {"start": 507.91999999999996, "end": 509.91999999999996, "text": " content representation of those two images"}, {"start": 510.44, "end": 512.68, "text": " We we can flatten those feature maps"}, {"start": 512.68, "end": 520.12, "text": " So each feature map becomes a row and this output matrix and now we need to drive those P and F matrix"}, {"start": 520.12, "end": 527.88, "text": " To be the same and we accomplish that using this loss, which is a simple MSE loss where you would just"}, {"start": 528.4, "end": 531.6, "text": " Take a element-wise subtraction and we do element-wise"}, {"start": 532.52, "end": 539.4, "text": " Squaring on those elements and we just try and drive that that loss to zero. I would try to minimize it"}, {"start": 540.72, "end": 543.0, "text": " And now for the really fun part"}, {"start": 543.0, "end": 548.04, "text": " And we're going to see what happens when we drive the loss down to zero"}, {"start": 548.04, "end": 550.7199999999999, "text": " And I'll just give you a couple of seconds to watch the animations"}, {"start": 555.64, "end": 558.68, "text": " So what you can see on the screen is on the left side"}, {"start": 558.68, "end": 563.48, "text": " You see what happens when the you see that the F matrix is getting closer to the P matrix"}, {"start": 563.48, "end": 569.9599999999999, "text": " Which is equivalent to the loss getting down to zero and on the right you can see the optimization procedure itself"}, {"start": 570.52, "end": 573.4399999999999, "text": " No, it's image becoming slowly becoming the input image"}, {"start": 573.44, "end": 577.72, "text": " The bottom animation is just the whole optimization procedure"}, {"start": 577.72, "end": 583.24, "text": " Whereas the upper animation is just the initial part of the optimization procedure in a slow-mo"}, {"start": 583.7600000000001, "end": 585.7600000000001, "text": " Because it happens really fast using"}, {"start": 586.2800000000001, "end": 595.12, "text": " LBFGS optimizer what you can see on the next screen is that the LBFGS is much more is much faster than the atom"}, {"start": 595.12, "end": 599.0, "text": " optimizer and in only 100 iterations LBFGS already"}, {"start": 599.0, "end": 602.92, "text": " Seems to be a morphing this noise image into content image"}, {"start": 602.92, "end": 608.36, "text": " Whereas atom is only just beginning to do that now for the second most important idea in this video"}, {"start": 608.36, "end": 614.56, "text": " And that's how do we capture the style of an image? So how do we find its style representation?"}, {"start": 615.28, "end": 619.28, "text": " So we have this style image input style image"}, {"start": 619.28, "end": 624.52, "text": " We feed it through this VGG net and we get a set of feature maps this time"}, {"start": 624.52, "end": 630.68, "text": " Taking those from starting from layer com 1 1 and going through layer com 5 1"}, {"start": 631.16, "end": 635.96, "text": " And what we do is we construct this feature space over these feature maps"}, {"start": 636.76, "end": 638.76, "text": " using something called"}, {"start": 639.0799999999999, "end": 640.6, "text": " gram transform"}, {"start": 640.6, "end": 645.36, "text": " So we create gram matrices out of those feature maps and"}, {"start": 645.68, "end": 650.48, "text": " The set of those gram matrices is what ultimately represents the style of the image"}, {"start": 650.48, "end": 658.2, "text": " Or let's call it the style representation and now you might ask what's a what's a gram matrix and that's a legit question"}, {"start": 659.0, "end": 665.08, "text": " So I tick a style image as an input I fed it through the VGG and from one of those layers"}, {"start": 665.6, "end": 670.8000000000001, "text": " I constructed a gram matrix, and this is how it looks like this is exactly how it looks like and"}, {"start": 671.32, "end": 676.44, "text": " It answers an important question and that's which feature maps are used in the VGG"}, {"start": 676.44, "end": 682.0, "text": " And it answers an important question and that's which feature maps tend to activate together"}, {"start": 682.5200000000001, "end": 686.0400000000001, "text": " we already saw how the feature maps look like and"}, {"start": 686.7600000000001, "end": 690.62, "text": " In one of the previous slides and now we have an answer to this question"}, {"start": 691.24, "end": 693.9200000000001, "text": " so it's a simple covariance matrix between"}, {"start": 694.84, "end": 696.6800000000001, "text": " different feature maps and"}, {"start": 696.6800000000001, "end": 703.08, "text": " the way we calculate an element in this matrix is by just doing a dot product between two feature maps and"}, {"start": 703.08, "end": 707.2800000000001, "text": " That just captures the the texture information as it turns out"}, {"start": 708.8000000000001, "end": 712.4000000000001, "text": " Let me give you some more intuition behind why gram matrix actually works"}, {"start": 713.72, "end": 717.64, "text": " So here's a hypothetical example where on the upper row we have a"}, {"start": 718.0400000000001, "end": 724.44, "text": " Hypothetical output three feature maps and on the bottom row the same thing or just for some other input image"}, {"start": 725.0, "end": 731.12, "text": " And if we would take element wise subtraction between those two rows would get a nonzero output"}, {"start": 731.12, "end": 733.68, "text": " which means we have a nonzero content loss"}, {"start": 734.72, "end": 739.96, "text": " Which means that the input images have different semantics right different content"}, {"start": 739.96, "end": 742.48, "text": " So the dog is upside down on the bottom row"}, {"start": 743.04, "end": 748.48, "text": " But on the other hand if you take a look on the right side, you'll see that the gram matrices are actually the same"}, {"start": 749.2, "end": 755.88, "text": " Which means that the two input images are stylized in the same manner, which is kind of true if you take a look at it and"}, {"start": 756.6, "end": 759.16, "text": " The style loss will be zero because of that"}, {"start": 759.16, "end": 762.7199999999999, "text": " So let's make it more explicit how we calculate the style loss"}, {"start": 763.24, "end": 766.92, "text": " So we have the input style image. We have the input noise image"}, {"start": 767.48, "end": 771.12, "text": " We feed them through a VGG. We'll get a set of feature maps here"}, {"start": 771.12, "end": 778.36, "text": " I'm only showing for simplicity just one set of feature maps we construct gram matrices over those feature maps"}, {"start": 778.68, "end": 781.76, "text": " And what we do is just a simple MSC loss again"}, {"start": 783.0, "end": 787.48, "text": " Which is just a element wise projection followed by element wise squared"}, {"start": 787.48, "end": 789.64, "text": " followed by element wise squaring and"}, {"start": 790.64, "end": 792.2, "text": " the final"}, {"start": 792.2, "end": 794.96, "text": " Style loss is actually just a weighted sum of those"}, {"start": 795.48, "end": 802.36, "text": " Terms for every layer in the network and that's it now. I'll do the same thing as a different condom"}, {"start": 802.36, "end": 808.2, "text": " Which let's see what happens when the style representation of the input noise image becomes the same as the style"}, {"start": 808.2, "end": 817.4000000000001, "text": " Representation of the input style image, and I'll give you a couple of seconds to just watch the animations watch closely"}, {"start": 818.0400000000001, "end": 820.5200000000001, "text": " And you can see there's a spike there so"}, {"start": 821.08, "end": 823.08, "text": " What happens is on the left side?"}, {"start": 823.2800000000001, "end": 829.76, "text": " G represents the set of gram matrices of the input noise image and a represents the set of"}, {"start": 829.96, "end": 834.96, "text": " Gram matrices of the input style image and as they are getting closer to each other"}, {"start": 834.96, "end": 839.52, "text": " On the right side you can see an animation where an input noise image"}, {"start": 840.24, "end": 842.6800000000001, "text": " Initially input noise image is slowly becoming"}, {"start": 844.08, "end": 849.48, "text": " Stylized it's capturing the style of this simple style image although. It's district disregarding the"}, {"start": 849.88, "end": 853.8000000000001, "text": " Semantics is just capturing the style. That's it and now putting it all together"}, {"start": 854.12, "end": 859.98, "text": " So the total loss is a way a combination of a common loss and the style loss"}, {"start": 859.98, "end": 865.24, "text": " And what it basically says is the following we want the input noise image you have the same"}, {"start": 865.72, "end": 869.48, "text": " style representation as the input style image and to have the same"}, {"start": 870.2, "end": 872.72, "text": " content representation as the input condom image and"}, {"start": 873.32, "end": 880.04, "text": " That objective might not be fully minimizable because a there does not exist a solution or b"}, {"start": 880.08, "end": 882.08, "text": " We cannot find a solution"}, {"start": 882.08, "end": 889.96, "text": " But still we'll get a visual appearance that we want and just take a look at the animation here"}, {"start": 889.96, "end": 893.08, "text": " And the line is slowly appearing"}, {"start": 894.08, "end": 899.88, "text": " In that style image and we are getting the composite image out that we and that was that was like the whole point of this"}, {"start": 899.88, "end": 904.2, "text": " Video so that's it for the second video if you like my content consider subscribing"}, {"start": 904.2, "end": 911.0600000000001, "text": " Gently push that like button and see you in the next video"}]