Advanced Theory | Neural Style Transfer #4
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

The 4th video in the neural style transfer series! 🎨
(I promise this will be the longest one 😅)

You'll learn:
✔️ Ideas behind how the NST field came to be
✔️ All the amazing follow-up work
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
00:35 - The Dawn of CNNs (aka AlexNet)
01:53 - Visualizing and Understanding Conv Nets
02:55 - Understanding Deep Image Representations by Inverting Them
05:00 - Texture Synthesis Using Convolutional Neural Networks
05:55 - The Birth of NST field (Gatys et al.)
07:00 - Feedforward methods (Johnson et al., Ulyanov et al.)
08:50 - Instance Normalization (Ulyanov et al.)
10:35 - Conditional Instance Normalization
11:50 - Controlling Perceptual Factors in NST (Gatys et al.)
14:15 - Demystifying Neural Style Transfer
15:15 - Arbitrary Style Transfer in Real-time with Adaptive IN
16:40 - Universal Style Transfer via Feature Transforms (WCT)
17:50 - Artistic Style Transfer for Videos (Ruder et al.)
19:05 - ReCoNet
19:25 - Going beyond: NST for spherical videos (VR), 3D models...
20:20 - Future challenges of NST field
21:20 - 0.5 million $ AI artwork

[Credits] Music:
https://www.youtube.com/watch?v=J2X5mJ3HDYE [NCS]

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition,
rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#neuralstyletransfer #deeplearning #ai
 In the last couple of videos we saw the basic theory of neural style transfer, we saw my implementation of the seminal work by Gatiss and his colleagues, and now we're going to put it all in the whole MSTL with them in a broader perspective and to see how it all came to be, as well as all the follow-up work that came afterwards, because it basically opened up a whole new research direction. So in this video we're going to cover the advanced neural style transfer theory, and the story starts in 2012. So there was this now already famous image genetic classification challenge, and in 2012 a new method was proposed using convolutional neural networks with the architecture called ILXNet, and it basically smashed all of the competitors, both in 2012 as well as normally in all the previous years. So for example in 2011, Sanchez and Peronin, they advised this method using really heavy mathematics, Fisher vectors and whatnot, and this method was twice as bad as ILXNet, even though it used so much math. You can see the quantum leap in 2012, where the error on the classification challenge went from 25.8 all the way down to 16.4. And we're going to focus on these three nets in this video. So ILXNet basically just sparked such a huge interest in CNNs, and ZFNet and VGG basically just explored the combinatorial space that was already set by the ILXNet. And people were generally interested in how CNN architecture worked. So there was this awesome paper in 2013 titled Visualizing and Understanding ComNets by the same guys who advised ZFNet. So they created these really cool visualizations that helped us better see which image structures tend to trigger certain feature maps. And you can see here on the screen that in the top left corner, this feature map gets triggered by when it gets dark faces in the input images. And on the top right, you can see that this feature map really likes some round objects, and bottom right, you can see that this feature map particularly likes spiral objects. VGG came afterwards, it just pretty much improved upon ILXNet and ZFNet by exploring the depth and the size of the kernel, the comkernel. And we still didn't quite understand how these deep codes work, or better said, what they've learned. So in 2014, this seminal work came along titled Understanding Deep Image Representations by Inverting Them. It was the first paper to propose this method of reconstructing input image from deep code, from feature maps, something we already know from previous videos. And let me just read it again. So we just do the optimization in the image space on a noise image. And you can see com1, for example, on the top left image, that we get a really detailed reconstruction if we try and invert the feature maps from those shallow layers. But if you go into deeper layers of the net and try to invert those codes, we get something like com4, that's an image underneath. And it's like more abstract. It still keeps the semantics of the image, but like concrete details are getting lost. This work was pretty much inception point for the creation of the Deep Dream algorithm by Google guys. And if you're not already familiar with it, Deep Dream just gives you all of these psychedelic-like images by exploiting what is called as Pareidolia effect. So what it does on a higher level is this. Whatever the network sees, like in certain feature maps, it just says, hey, whatever you see, give me more of it. The implementation is as simple as just maximizing the feature response at a certain layer by doing, say, gradient ascent and not gradient descent. And that's equivalent to saying, give me more of what you see. But more important for our story, it was the first main ingredient for the inception of neural style transfer algorithm. The second main ingredient also came from the creator of NST, Leon Gatiss, in the work titled Texture Synthesis Using ComNet. And here he just exploited the rich feature representation of the VGG network to create these awesome textures that you can see on the screen. And this is basically the same thing we did in the last video. It is important to appreciate that this work also did not come out of the blue. So the conceptual framework of building up some summary statistics over certain filter responses was already in place. So, for example, in Portilla and Simoncelli's work, instead of using feature filter responses of VGG net, they used the filter responses of a linear filter bank. And instead of using Grammatrix that captures the correlations between feature maps, they use a carefully chosen set of summary statistics. And finally, combining the previous work of reconstructing images from deep codes, which basically gives us the content portion of the stylized image and combining that with Gatiss' texture synthesis work, where conceptually transferring style is equivalent to transferring texture, so this part gives us the style portion of the stylized image. We finally get the final algorithm. It's just interesting how connecting a couple of dots created a surge of research in this new direction. Lots of follow-up work came after the original algorithm was advised back in 2015. And what is interesting is that there is this interesting relationship that ties all of these algorithms that came afterwards. And that's this three-way trade-off between speed, quality, and flexibility in the number of styles that the algorithm can produce. And that will become clearer what that means a bit later in the video. The original algorithm was pretty high quality, infinite flexibility in the sense you can transfer any style, but really slow. So let's see how we can improve the speed portion. So the main idea is this. Instead of using the optimization algorithm, let's just pass in the image, do a feed-forward pass, and get a stylized image out. And there were basically two independent papers that implemented this idea back in March 2016, and those were by Johnson and by Iuliano. And I'll show the Johnson's method here because it's conceptually simpler, a bit higher quality, but a bit lower speed. So the method goes like this. So we're optimizing this image transform net's weights and not the pixels in the image space. The loss is the same as in Gatiss work, and it gets defined by the deep net. And I always find that really interesting. And it gets trained on the MSCoco dataset. So we iterate through images in the dataset and the style loss is fixed, but the content loss is specific for every single image in the dataset. And by doing so, the net learns to minimize the style on arbitrary input image. Let's see how it ranks against the three-way tradeoff. So it's still the fastest implementation out there. It's got the lowest possible flexibility. It supports only one style. And the quality, you can see the graphs here and let's focus on the leftmost graph because it's just for the lowest res input image. And in the intersection of the blue and green curves, that defines the place where the loss is the same for the two methods, for the Gatiss and Johnson's methods. And we can see that happens around 80th iteration of LBFGS optimizer, which means that the quality of this method is the same as Gatiss after 80 LBFGS iterations. Now, there's a reason I mentioned Ulyanov here. He basically unlocked the quality and flexibility for these FID4 methods by introducing the concept of instance normalization. Instance normalization is really similar to batch normalization, which was advised a year and a half before by Christian Zagady and Sergey Iofi from Google. And the only difference is, let's say, the space over which you calculate the statistics. So whereas batch normalization use a particular feature map from every single training example, the instance normalization just uses a single feature map. And you can see the image here where the spatial dimensions of the feature map, the H and W, are collapsed in a single dimension. And N is the mini batch size that the instance normalization is using just a single training example to figure out those statistics. And let me just reiterate it. When I say statistics, I mean finding the mean and variance of the distribution and using those to normalize the distribution, making it univariance and zero mean. And later applying those FN params, those betas and gammas to just keep the original representation power of the network. Now, if you've never heard about better realization, this will probably sound like rubbish to you. And I'd usually suggest reading the original paper. But this time the paper is really vague and the visualizations are really bad. So I just suggest using either medium, some either medium blog or towards data science blog. And I'll link some of those in the description. Let's see these normalization layers in action. So if you apply those in the generated network, we get these results. And in the bottom row, you can see that the institutionalization achieves greater quality. And I already mentioned that institutionalization unlock greater quality and bigger flexibility. Also, the first paper to exploit greater flexibility was this conditional institutionalization paper. And they achieved 32 styles where there wasn't a hard limit. It was just that the number of parameters grows linearly if you want to add more styles. The main idea goes like this. So we do the same thing as an institutionalization and that's normalized distribution, making it univariance and zero mean. And then instead of using a single pair of betas and gammas as an institutionalization, every single style has this pair associated with it. And this simple idea enables us to create multiple stylized images using multiple styles. And you can see here that interpolating those different styles, we can get a whole like continuous space of new stylized images. And it's really surprising that using only two single parameters, we can define a completely new style. So we've seen some really high quality methods like the original gages method. We've seen some really fast methods like Johnson's method. And we've seen some semi-flexible methods like these this conditional institutionalization. So what else do we want from our NST algorithm? And the answer is control. You usually don't have the control of what the network or the algorithm outputs. And you want to control stuff like space in the sense we slightly apply to which portion of the which region of the image. And then you want to control whether you take the color from the count image or take it from the style image. And you want to also have control over which brush strokes to use on the coarse scale and which ones to use on the fine scale. So let's take a look at the spatial control a bit deeper. So the idea goes like this. Let's take the sky region of the style image, which is defined by the black pixels of its corresponding segmentation mask. You can see it on the top right corner of the image and apply to the sky region of the content image, which is defined by the black pixels of its corresponding segmentation mask. And this time, let's take this whole style image and apply to the non-sky region of the content image, which is defined by the white pixels of its corresponding segmentation mask. This mixing of styles doesn't really happen in the image space by just combining those images with the segmentation mask. It happens in the feature space where they use some morphological operators such as erosion to get those nicely blended together. And when it comes to color control, first, why? Well, sometimes you just get an output image which you don't like, like this one. Now for the how portion. Well, one method is to do this. You take the content and style images, you transform them into some color space where the color information and the intensity information is separable. And what you do is you take the luminance components of style and content images, you do the style transfer, and then you take the color channels from the content image and just concatenate those with the output and you get the final image. And that's the one you see under D. Now, controlling scale is really simple. What you do is you take a fine scale brushstrokes from one painting, you combine those with a core scale angular geometric shapes from another style image, and you produce a new style image. And then you just use that one in NSD in a classical NSD procedure to get the image under E. And just be aware that this is something useful to have enough for the fun part. So up until now, we consider those grand matrices to be some kind of unnatural law, like we had to match those in order to transfer style. And as this paper shows, demystifying neural style transfer, matching those grand matrices is nothing but like minimizing this maximum mean distribution with a polynomial kernel. That means we can use other kernels like linear or Gaussian kernel to achieve style transfer. And as it says right here, this reveals that neural style transfer is intrinsically a process of distribution alignment of the neural activations between images, which means we basically just need to align those distributions in order to transfer style. And there are various ways to do that. And one important that will be important for us is this batch normalization statistics. And we already saw a hint for that in the conditional instance normalization paper. Now, this work took it even further and achieved infinite flexibility. I can transfer any style possible and it does that the following way. So you take the image, you pass it through the feed forward map and you take a specific feature map. You just normalize it by finding its mean and variance. And then you do the same thing for the style image. You find the same feature map. You find the mean and the variance and you just take those mean and variance parameters and apply them to the content feature map. You pass it through the decoder and you get a stylized image out. So no learnable parameters this time. You just pass two single parameters and you achieve style transfer. Let's see it once more on the screen. So you take the content feature map, you normalize it by finding its mean and variance. And then you find the mean and variance for the style image and you just reapply to the content image and you get a stylized image. Now, the good thing is that those affine parameters, those betas and gammas don't need to be learned anymore, such as in batch normalization. Also in instance normalization, we had to learn those and in conditional instance normalization. Here, they are not a learnable parameters. But the bad thing is that the coder still has to be trained. So you need a finite set of style images upon which to train this decoder. And that means it won't be it won't perform as good for unseen style images. The follow up work fixed this problem, achieving truly infinite flexibility in the sense of number of styles it can apply, although it had you had to sacrifice the quality a little bit as per three way trade off. And the method works like this. So you just train a simple image reconstruction or the encoder and you insert this WCT block. And let's see what what it does. So it does the whitening on the content features by just by just basically figuring out the composition of those content features, applying a couple of linear transformations and ending up with a grand matrix that's uncorrelated, meaning only the values on the diagonal have once and everything else is zero. Just don't do all maths here. Just try and follow along. And then you just find the same I can decomposition. But this time for style features apply a couple of linear transformations and you end up with content features that have the same same matrix as the style image, which is something we always did, but this time without any learning, any training. So we saw the evolution of methods for static images and in parallel people were advising new methods for videos. And the only additional problem that these methods have to solve is how to keep the temporal coherence between frames. And as you can see here, we've got some original frames in the style image. And if you just do a dummy like applying the NST per per single frame, there's going to be a lot of flickering, a lot of inconsistency between different frames because every single time you run it, the image gets a different local optimum. Whereas when you apply the with temporal constraint, you can see that the images are much more smoother and consistent between frames. The way we achieve this temporal consistency is the following. So we take this green frame, the previous frame, and we forward warp it using the optical flow information. And we take this red frame, the next frame, and we penalize it for deviating from the green frame on those locations, on those pixels where the optical flow was stable. There was no disocclusion happening. This method used a slow optimization procedure. So needless to say, it was painfully slow. And the follow up work just did the same thing as for static images. It just transferred this problem into a feed forward network. And I've actually used this very same model in the beginning of the series to create the very first clip. It's called Reconet. And this is the clip you created. So in the beginning, it's temporal inconsistent and it becomes temporally consistent and smooth. So we saw a neural style transfer for static images, we saw for videos. And the thing with this NST field is that people are trying to apply it everywhere. So we've got NST for spherical images and videos that can be used in VR. There is this concept of transferring style to 3D models where the artist just needs to paint this sphere in the top right corner. And it gets transferred to the 3D model. There is this photorealistic neural style transfer where you basically only transfer the color information onto the content image. There is also a neural style transfer for audio and style aware content loss, etc., etc. So it would be really valuable to share how this whole field just evolved, how people were trying to connect various dots and build upon each other's work. And that's just how research looks like. The field is nowhere near having all problems sorted out. There are still a lot of challenges out there. One of those is static evaluation. So there is still not there doesn't exist some numerical method which can help us compare different NST methods and say, hey, this is better. This one is worse. We're still using those side by side subjective visual comparisons and different user studies to figure out which method is better. Also, there is no standard benchmark image set, meaning everybody is using their own style images. Everybody is using their own content images. And it's kind of hard to just wrap your head around and compare different methods. In other challenges, this representation disentangling. So we saw some efforts like the controlling those perceptual factors like scale, like space and color. It would be really nice if we had some latent space representation where we could just tweak certain dimension and get all of those perceptual factors changing the way we want them to change. And I want to wrap this up with a fun fact. And do you see the image on the right there? Could you guess it was it was sold in an auction? Could you guess the price tag? It was almost half a million dollars. And the thing is, it wasn't even created by neural style transfer algorithm. It was created by GANs. You can see the equation down there. But it's just so interesting that we are living in an age where there is this interesting dynamics going on between art and tech. And I think it's really a great time to be alive. So I really hope you found this video valuable. If you're new to my channel, consider subscribing and sharing. And there's awesome new content coming up. So stay tuned and see in the next video.
[{"start": 0.0, "end": 12.0, "text": " In the last couple of videos we saw the basic theory of neural style transfer, we saw my implementation of the seminal work by Gatiss and his colleagues,"}, {"start": 12.0, "end": 20.0, "text": " and now we're going to put it all in the whole MSTL with them in a broader perspective and to see how it all came to be,"}, {"start": 20.0, "end": 28.0, "text": " as well as all the follow-up work that came afterwards, because it basically opened up a whole new research direction."}, {"start": 28.0, "end": 37.0, "text": " So in this video we're going to cover the advanced neural style transfer theory, and the story starts in 2012."}, {"start": 37.0, "end": 47.0, "text": " So there was this now already famous image genetic classification challenge, and in 2012 a new method was proposed using convolutional neural networks"}, {"start": 47.0, "end": 57.0, "text": " with the architecture called ILXNet, and it basically smashed all of the competitors, both in 2012 as well as normally in all the previous years."}, {"start": 57.0, "end": 67.0, "text": " So for example in 2011, Sanchez and Peronin, they advised this method using really heavy mathematics, Fisher vectors and whatnot,"}, {"start": 67.0, "end": 77.0, "text": " and this method was twice as bad as ILXNet, even though it used so much math."}, {"start": 77.0, "end": 87.0, "text": " You can see the quantum leap in 2012, where the error on the classification challenge went from 25.8 all the way down to 16.4."}, {"start": 87.0, "end": 92.0, "text": " And we're going to focus on these three nets in this video."}, {"start": 92.0, "end": 105.0, "text": " So ILXNet basically just sparked such a huge interest in CNNs, and ZFNet and VGG basically just explored the combinatorial space that was already set by the ILXNet."}, {"start": 105.0, "end": 112.0, "text": " And people were generally interested in how CNN architecture worked."}, {"start": 112.0, "end": 123.0, "text": " So there was this awesome paper in 2013 titled Visualizing and Understanding ComNets by the same guys who advised ZFNet."}, {"start": 123.0, "end": 133.0, "text": " So they created these really cool visualizations that helped us better see which image structures tend to trigger certain feature maps."}, {"start": 133.0, "end": 147.0, "text": " And you can see here on the screen that in the top left corner, this feature map gets triggered by when it gets dark faces in the input images."}, {"start": 147.0, "end": 160.0, "text": " And on the top right, you can see that this feature map really likes some round objects, and bottom right, you can see that this feature map particularly likes spiral objects."}, {"start": 160.0, "end": 175.0, "text": " VGG came afterwards, it just pretty much improved upon ILXNet and ZFNet by exploring the depth and the size of the kernel, the comkernel."}, {"start": 175.0, "end": 184.0, "text": " And we still didn't quite understand how these deep codes work, or better said, what they've learned."}, {"start": 184.0, "end": 194.0, "text": " So in 2014, this seminal work came along titled Understanding Deep Image Representations by Inverting Them."}, {"start": 194.0, "end": 205.0, "text": " It was the first paper to propose this method of reconstructing input image from deep code, from feature maps, something we already know from previous videos."}, {"start": 205.0, "end": 212.0, "text": " And let me just read it again. So we just do the optimization in the image space on a noise image."}, {"start": 212.0, "end": 229.0, "text": " And you can see com1, for example, on the top left image, that we get a really detailed reconstruction if we try and invert the feature maps from those shallow layers."}, {"start": 229.0, "end": 237.0, "text": " But if you go into deeper layers of the net and try to invert those codes, we get something like com4, that's an image underneath."}, {"start": 237.0, "end": 246.0, "text": " And it's like more abstract. It still keeps the semantics of the image, but like concrete details are getting lost."}, {"start": 246.0, "end": 252.0, "text": " This work was pretty much inception point for the creation of the Deep Dream algorithm by Google guys."}, {"start": 252.0, "end": 263.0, "text": " And if you're not already familiar with it, Deep Dream just gives you all of these psychedelic-like images by exploiting what is called as Pareidolia effect."}, {"start": 263.0, "end": 273.0, "text": " So what it does on a higher level is this. Whatever the network sees, like in certain feature maps, it just says, hey, whatever you see, give me more of it."}, {"start": 273.0, "end": 284.0, "text": " The implementation is as simple as just maximizing the feature response at a certain layer by doing, say, gradient ascent and not gradient descent."}, {"start": 284.0, "end": 288.0, "text": " And that's equivalent to saying, give me more of what you see."}, {"start": 288.0, "end": 297.0, "text": " But more important for our story, it was the first main ingredient for the inception of neural style transfer algorithm."}, {"start": 297.0, "end": 305.0, "text": " The second main ingredient also came from the creator of NST, Leon Gatiss, in the work titled Texture Synthesis Using ComNet."}, {"start": 305.0, "end": 316.0, "text": " And here he just exploited the rich feature representation of the VGG network to create these awesome textures that you can see on the screen."}, {"start": 316.0, "end": 319.0, "text": " And this is basically the same thing we did in the last video."}, {"start": 319.0, "end": 323.0, "text": " It is important to appreciate that this work also did not come out of the blue."}, {"start": 323.0, "end": 333.0, "text": " So the conceptual framework of building up some summary statistics over certain filter responses was already in place."}, {"start": 333.0, "end": 344.0, "text": " So, for example, in Portilla and Simoncelli's work, instead of using feature filter responses of VGG net, they used the filter responses of a linear filter bank."}, {"start": 344.0, "end": 352.0, "text": " And instead of using Grammatrix that captures the correlations between feature maps, they use a carefully chosen set of summary statistics."}, {"start": 352.0, "end": 367.0, "text": " And finally, combining the previous work of reconstructing images from deep codes, which basically gives us the content portion of the stylized image and combining that with Gatiss' texture synthesis work,"}, {"start": 367.0, "end": 376.0, "text": " where conceptually transferring style is equivalent to transferring texture, so this part gives us the style portion of the stylized image."}, {"start": 376.0, "end": 379.0, "text": " We finally get the final algorithm."}, {"start": 379.0, "end": 386.0, "text": " It's just interesting how connecting a couple of dots created a surge of research in this new direction."}, {"start": 386.0, "end": 392.0, "text": " Lots of follow-up work came after the original algorithm was advised back in 2015."}, {"start": 392.0, "end": 399.0, "text": " And what is interesting is that there is this interesting relationship that ties all of these algorithms that came afterwards."}, {"start": 399.0, "end": 408.0, "text": " And that's this three-way trade-off between speed, quality, and flexibility in the number of styles that the algorithm can produce."}, {"start": 408.0, "end": 412.0, "text": " And that will become clearer what that means a bit later in the video."}, {"start": 412.0, "end": 420.0, "text": " The original algorithm was pretty high quality, infinite flexibility in the sense you can transfer any style, but really slow."}, {"start": 420.0, "end": 423.0, "text": " So let's see how we can improve the speed portion."}, {"start": 423.0, "end": 432.0, "text": " So the main idea is this. Instead of using the optimization algorithm, let's just pass in the image, do a feed-forward pass, and get a stylized image out."}, {"start": 432.0, "end": 441.0, "text": " And there were basically two independent papers that implemented this idea back in March 2016, and those were by Johnson and by Iuliano."}, {"start": 441.0, "end": 448.0, "text": " And I'll show the Johnson's method here because it's conceptually simpler, a bit higher quality, but a bit lower speed."}, {"start": 448.0, "end": 457.0, "text": " So the method goes like this. So we're optimizing this image transform net's weights and not the pixels in the image space."}, {"start": 457.0, "end": 462.0, "text": " The loss is the same as in Gatiss work, and it gets defined by the deep net."}, {"start": 462.0, "end": 468.0, "text": " And I always find that really interesting. And it gets trained on the MSCoco dataset."}, {"start": 468.0, "end": 479.0, "text": " So we iterate through images in the dataset and the style loss is fixed, but the content loss is specific for every single image in the dataset."}, {"start": 479.0, "end": 484.0, "text": " And by doing so, the net learns to minimize the style on arbitrary input image."}, {"start": 484.0, "end": 490.0, "text": " Let's see how it ranks against the three-way tradeoff. So it's still the fastest implementation out there."}, {"start": 490.0, "end": 494.0, "text": " It's got the lowest possible flexibility. It supports only one style."}, {"start": 494.0, "end": 502.0, "text": " And the quality, you can see the graphs here and let's focus on the leftmost graph because it's just for the lowest res input image."}, {"start": 502.0, "end": 511.0, "text": " And in the intersection of the blue and green curves, that defines the place where the loss is the same for the two methods, for the Gatiss and Johnson's methods."}, {"start": 511.0, "end": 522.0, "text": " And we can see that happens around 80th iteration of LBFGS optimizer, which means that the quality of this method is the same as Gatiss after 80 LBFGS iterations."}, {"start": 522.0, "end": 533.0, "text": " Now, there's a reason I mentioned Ulyanov here. He basically unlocked the quality and flexibility for these FID4 methods by introducing the concept of instance normalization."}, {"start": 533.0, "end": 542.0, "text": " Instance normalization is really similar to batch normalization, which was advised a year and a half before by Christian Zagady and Sergey Iofi from Google."}, {"start": 542.0, "end": 547.0, "text": " And the only difference is, let's say, the space over which you calculate the statistics."}, {"start": 547.0, "end": 556.0, "text": " So whereas batch normalization use a particular feature map from every single training example, the instance normalization just uses a single feature map."}, {"start": 556.0, "end": 564.0, "text": " And you can see the image here where the spatial dimensions of the feature map, the H and W, are collapsed in a single dimension."}, {"start": 564.0, "end": 572.0, "text": " And N is the mini batch size that the instance normalization is using just a single training example to figure out those statistics."}, {"start": 572.0, "end": 585.0, "text": " And let me just reiterate it. When I say statistics, I mean finding the mean and variance of the distribution and using those to normalize the distribution, making it univariance and zero mean."}, {"start": 585.0, "end": 592.0, "text": " And later applying those FN params, those betas and gammas to just keep the original representation power of the network."}, {"start": 592.0, "end": 602.0, "text": " Now, if you've never heard about better realization, this will probably sound like rubbish to you. And I'd usually suggest reading the original paper."}, {"start": 602.0, "end": 606.0, "text": " But this time the paper is really vague and the visualizations are really bad."}, {"start": 606.0, "end": 613.0, "text": " So I just suggest using either medium, some either medium blog or towards data science blog."}, {"start": 613.0, "end": 617.0, "text": " And I'll link some of those in the description. Let's see these normalization layers in action."}, {"start": 617.0, "end": 622.0, "text": " So if you apply those in the generated network, we get these results."}, {"start": 622.0, "end": 627.0, "text": " And in the bottom row, you can see that the institutionalization achieves greater quality."}, {"start": 627.0, "end": 633.0, "text": " And I already mentioned that institutionalization unlock greater quality and bigger flexibility."}, {"start": 633.0, "end": 642.0, "text": " Also, the first paper to exploit greater flexibility was this conditional institutionalization paper."}, {"start": 642.0, "end": 647.0, "text": " And they achieved 32 styles where there wasn't a hard limit."}, {"start": 647.0, "end": 651.0, "text": " It was just that the number of parameters grows linearly if you want to add more styles."}, {"start": 651.0, "end": 661.0, "text": " The main idea goes like this. So we do the same thing as an institutionalization and that's normalized distribution, making it univariance and zero mean."}, {"start": 661.0, "end": 672.0, "text": " And then instead of using a single pair of betas and gammas as an institutionalization, every single style has this pair associated with it."}, {"start": 672.0, "end": 679.0, "text": " And this simple idea enables us to create multiple stylized images using multiple styles."}, {"start": 679.0, "end": 687.0, "text": " And you can see here that interpolating those different styles, we can get a whole like continuous space of new stylized images."}, {"start": 687.0, "end": 694.0, "text": " And it's really surprising that using only two single parameters, we can define a completely new style."}, {"start": 694.0, "end": 698.0, "text": " So we've seen some really high quality methods like the original gages method."}, {"start": 698.0, "end": 701.0, "text": " We've seen some really fast methods like Johnson's method."}, {"start": 701.0, "end": 707.0, "text": " And we've seen some semi-flexible methods like these this conditional institutionalization."}, {"start": 707.0, "end": 712.0, "text": " So what else do we want from our NST algorithm? And the answer is control."}, {"start": 712.0, "end": 717.0, "text": " You usually don't have the control of what the network or the algorithm outputs."}, {"start": 717.0, "end": 726.0, "text": " And you want to control stuff like space in the sense we slightly apply to which portion of the which region of the image."}, {"start": 726.0, "end": 732.0, "text": " And then you want to control whether you take the color from the count image or take it from the style image."}, {"start": 732.0, "end": 740.0, "text": " And you want to also have control over which brush strokes to use on the coarse scale and which ones to use on the fine scale."}, {"start": 740.0, "end": 744.0, "text": " So let's take a look at the spatial control a bit deeper."}, {"start": 744.0, "end": 753.0, "text": " So the idea goes like this. Let's take the sky region of the style image, which is defined by the black pixels of its corresponding segmentation mask."}, {"start": 753.0, "end": 763.0, "text": " You can see it on the top right corner of the image and apply to the sky region of the content image, which is defined by the black pixels of its corresponding segmentation mask."}, {"start": 763.0, "end": 774.0, "text": " And this time, let's take this whole style image and apply to the non-sky region of the content image, which is defined by the white pixels of its corresponding segmentation mask."}, {"start": 774.0, "end": 781.0, "text": " This mixing of styles doesn't really happen in the image space by just combining those images with the segmentation mask."}, {"start": 781.0, "end": 789.0, "text": " It happens in the feature space where they use some morphological operators such as erosion to get those nicely blended together."}, {"start": 789.0, "end": 798.0, "text": " And when it comes to color control, first, why? Well, sometimes you just get an output image which you don't like, like this one."}, {"start": 798.0, "end": 802.0, "text": " Now for the how portion. Well, one method is to do this."}, {"start": 802.0, "end": 812.0, "text": " You take the content and style images, you transform them into some color space where the color information and the intensity information is separable."}, {"start": 812.0, "end": 827.0, "text": " And what you do is you take the luminance components of style and content images, you do the style transfer, and then you take the color channels from the content image and just concatenate those with the output and you get the final image."}, {"start": 827.0, "end": 832.0, "text": " And that's the one you see under D. Now, controlling scale is really simple."}, {"start": 832.0, "end": 845.0, "text": " What you do is you take a fine scale brushstrokes from one painting, you combine those with a core scale angular geometric shapes from another style image, and you produce a new style image."}, {"start": 845.0, "end": 851.0, "text": " And then you just use that one in NSD in a classical NSD procedure to get the image under E."}, {"start": 851.0, "end": 855.0, "text": " And just be aware that this is something useful to have enough for the fun part."}, {"start": 855.0, "end": 863.0, "text": " So up until now, we consider those grand matrices to be some kind of unnatural law, like we had to match those in order to transfer style."}, {"start": 863.0, "end": 877.0, "text": " And as this paper shows, demystifying neural style transfer, matching those grand matrices is nothing but like minimizing this maximum mean distribution with a polynomial kernel."}, {"start": 877.0, "end": 884.0, "text": " That means we can use other kernels like linear or Gaussian kernel to achieve style transfer."}, {"start": 884.0, "end": 899.0, "text": " And as it says right here, this reveals that neural style transfer is intrinsically a process of distribution alignment of the neural activations between images, which means we basically just need to align those distributions in order to transfer style."}, {"start": 899.0, "end": 908.0, "text": " And there are various ways to do that. And one important that will be important for us is this batch normalization statistics."}, {"start": 908.0, "end": 913.0, "text": " And we already saw a hint for that in the conditional instance normalization paper."}, {"start": 913.0, "end": 918.0, "text": " Now, this work took it even further and achieved infinite flexibility."}, {"start": 918.0, "end": 923.0, "text": " I can transfer any style possible and it does that the following way."}, {"start": 923.0, "end": 929.0, "text": " So you take the image, you pass it through the feed forward map and you take a specific feature map."}, {"start": 929.0, "end": 933.0, "text": " You just normalize it by finding its mean and variance."}, {"start": 933.0, "end": 936.0, "text": " And then you do the same thing for the style image."}, {"start": 936.0, "end": 938.0, "text": " You find the same feature map."}, {"start": 938.0, "end": 945.0, "text": " You find the mean and the variance and you just take those mean and variance parameters and apply them to the content feature map."}, {"start": 945.0, "end": 949.0, "text": " You pass it through the decoder and you get a stylized image out."}, {"start": 949.0, "end": 951.0, "text": " So no learnable parameters this time."}, {"start": 951.0, "end": 955.0, "text": " You just pass two single parameters and you achieve style transfer."}, {"start": 955.0, "end": 957.0, "text": " Let's see it once more on the screen."}, {"start": 957.0, "end": 961.0, "text": " So you take the content feature map, you normalize it by finding its mean and variance."}, {"start": 961.0, "end": 969.0, "text": " And then you find the mean and variance for the style image and you just reapply to the content image and you get a stylized image."}, {"start": 969.0, "end": 978.0, "text": " Now, the good thing is that those affine parameters, those betas and gammas don't need to be learned anymore, such as in batch normalization."}, {"start": 978.0, "end": 983.0, "text": " Also in instance normalization, we had to learn those and in conditional instance normalization."}, {"start": 983.0, "end": 986.0, "text": " Here, they are not a learnable parameters."}, {"start": 986.0, "end": 989.0, "text": " But the bad thing is that the coder still has to be trained."}, {"start": 989.0, "end": 993.0, "text": " So you need a finite set of style images upon which to train this decoder."}, {"start": 993.0, "end": 998.0, "text": " And that means it won't be it won't perform as good for unseen style images."}, {"start": 998.0, "end": 1008.0, "text": " The follow up work fixed this problem, achieving truly infinite flexibility in the sense of number of styles it can apply,"}, {"start": 1008.0, "end": 1013.0, "text": " although it had you had to sacrifice the quality a little bit as per three way trade off."}, {"start": 1013.0, "end": 1014.0, "text": " And the method works like this."}, {"start": 1014.0, "end": 1021.0, "text": " So you just train a simple image reconstruction or the encoder and you insert this WCT block."}, {"start": 1021.0, "end": 1023.0, "text": " And let's see what what it does."}, {"start": 1023.0, "end": 1034.0, "text": " So it does the whitening on the content features by just by just basically figuring out the composition of those content features,"}, {"start": 1034.0, "end": 1039.0, "text": " applying a couple of linear transformations and ending up with a grand matrix that's uncorrelated,"}, {"start": 1039.0, "end": 1044.0, "text": " meaning only the values on the diagonal have once and everything else is zero."}, {"start": 1044.0, "end": 1046.0, "text": " Just don't do all maths here."}, {"start": 1046.0, "end": 1047.0, "text": " Just try and follow along."}, {"start": 1047.0, "end": 1051.0, "text": " And then you just find the same I can decomposition."}, {"start": 1051.0, "end": 1062.0, "text": " But this time for style features apply a couple of linear transformations and you end up with content features that have the same same matrix as the style image,"}, {"start": 1062.0, "end": 1068.0, "text": " which is something we always did, but this time without any learning, any training."}, {"start": 1068.0, "end": 1075.0, "text": " So we saw the evolution of methods for static images and in parallel people were advising new methods for videos."}, {"start": 1075.0, "end": 1082.0, "text": " And the only additional problem that these methods have to solve is how to keep the temporal coherence between frames."}, {"start": 1082.0, "end": 1087.0, "text": " And as you can see here, we've got some original frames in the style image."}, {"start": 1087.0, "end": 1095.0, "text": " And if you just do a dummy like applying the NST per per single frame, there's going to be a lot of flickering,"}, {"start": 1095.0, "end": 1104.0, "text": " a lot of inconsistency between different frames because every single time you run it, the image gets a different local optimum."}, {"start": 1104.0, "end": 1111.0, "text": " Whereas when you apply the with temporal constraint, you can see that the images are much more smoother and consistent between frames."}, {"start": 1111.0, "end": 1114.0, "text": " The way we achieve this temporal consistency is the following."}, {"start": 1114.0, "end": 1122.0, "text": " So we take this green frame, the previous frame, and we forward warp it using the optical flow information."}, {"start": 1122.0, "end": 1133.0, "text": " And we take this red frame, the next frame, and we penalize it for deviating from the green frame on those locations,"}, {"start": 1133.0, "end": 1136.0, "text": " on those pixels where the optical flow was stable."}, {"start": 1136.0, "end": 1139.0, "text": " There was no disocclusion happening."}, {"start": 1139.0, "end": 1142.0, "text": " This method used a slow optimization procedure."}, {"start": 1142.0, "end": 1144.0, "text": " So needless to say, it was painfully slow."}, {"start": 1144.0, "end": 1148.0, "text": " And the follow up work just did the same thing as for static images."}, {"start": 1148.0, "end": 1151.0, "text": " It just transferred this problem into a feed forward network."}, {"start": 1151.0, "end": 1157.0, "text": " And I've actually used this very same model in the beginning of the series to create the very first clip."}, {"start": 1157.0, "end": 1160.0, "text": " It's called Reconet. And this is the clip you created."}, {"start": 1160.0, "end": 1166.0, "text": " So in the beginning, it's temporal inconsistent and it becomes temporally consistent and smooth."}, {"start": 1166.0, "end": 1171.0, "text": " So we saw a neural style transfer for static images, we saw for videos."}, {"start": 1171.0, "end": 1175.0, "text": " And the thing with this NST field is that people are trying to apply it everywhere."}, {"start": 1175.0, "end": 1180.0, "text": " So we've got NST for spherical images and videos that can be used in VR."}, {"start": 1180.0, "end": 1191.0, "text": " There is this concept of transferring style to 3D models where the artist just needs to paint this sphere in the top right corner."}, {"start": 1191.0, "end": 1193.0, "text": " And it gets transferred to the 3D model."}, {"start": 1193.0, "end": 1201.0, "text": " There is this photorealistic neural style transfer where you basically only transfer the color information onto the content image."}, {"start": 1201.0, "end": 1208.0, "text": " There is also a neural style transfer for audio and style aware content loss, etc., etc."}, {"start": 1208.0, "end": 1212.0, "text": " So it would be really valuable to share how this whole field just evolved,"}, {"start": 1212.0, "end": 1217.0, "text": " how people were trying to connect various dots and build upon each other's work."}, {"start": 1217.0, "end": 1219.0, "text": " And that's just how research looks like."}, {"start": 1219.0, "end": 1222.0, "text": " The field is nowhere near having all problems sorted out."}, {"start": 1222.0, "end": 1224.0, "text": " There are still a lot of challenges out there."}, {"start": 1224.0, "end": 1226.0, "text": " One of those is static evaluation."}, {"start": 1226.0, "end": 1235.0, "text": " So there is still not there doesn't exist some numerical method which can help us compare different NST methods and say, hey, this is better."}, {"start": 1235.0, "end": 1236.0, "text": " This one is worse."}, {"start": 1236.0, "end": 1245.0, "text": " We're still using those side by side subjective visual comparisons and different user studies to figure out which method is better."}, {"start": 1245.0, "end": 1251.0, "text": " Also, there is no standard benchmark image set, meaning everybody is using their own style images."}, {"start": 1251.0, "end": 1253.0, "text": " Everybody is using their own content images."}, {"start": 1253.0, "end": 1257.0, "text": " And it's kind of hard to just wrap your head around and compare different methods."}, {"start": 1257.0, "end": 1260.0, "text": " In other challenges, this representation disentangling."}, {"start": 1260.0, "end": 1267.0, "text": " So we saw some efforts like the controlling those perceptual factors like scale, like space and color."}, {"start": 1267.0, "end": 1278.0, "text": " It would be really nice if we had some latent space representation where we could just tweak certain dimension and get all of those perceptual factors changing the way we want them to change."}, {"start": 1278.0, "end": 1281.0, "text": " And I want to wrap this up with a fun fact."}, {"start": 1281.0, "end": 1283.0, "text": " And do you see the image on the right there?"}, {"start": 1283.0, "end": 1286.0, "text": " Could you guess it was it was sold in an auction?"}, {"start": 1286.0, "end": 1287.0, "text": " Could you guess the price tag?"}, {"start": 1287.0, "end": 1290.0, "text": " It was almost half a million dollars."}, {"start": 1290.0, "end": 1294.0, "text": " And the thing is, it wasn't even created by neural style transfer algorithm."}, {"start": 1294.0, "end": 1295.0, "text": " It was created by GANs."}, {"start": 1295.0, "end": 1297.0, "text": " You can see the equation down there."}, {"start": 1297.0, "end": 1303.0, "text": " But it's just so interesting that we are living in an age where there is this interesting dynamics going on between art and tech."}, {"start": 1303.0, "end": 1306.0, "text": " And I think it's really a great time to be alive."}, {"start": 1306.0, "end": 1309.0, "text": " So I really hope you found this video valuable."}, {"start": 1309.0, "end": 1312.0, "text": " If you're new to my channel, consider subscribing and sharing."}, {"start": 1312.0, "end": 1315.0, "text": " And there's awesome new content coming up."}, {"start": 1315.0, "end": 1318.0, "text": " So stay tuned and see in the next video."}]