GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

In this video I cover the "GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation" paper.

They managed to increase the diversity and stability of anime avatar generation and as a side benefit - they can also apply it to videos without any additional temporal constraints!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2106.06561
✅ GitHub: https://github.com/mchong6/GANsNRoses
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable:
00:00 High-level overview of the paper
04:00 Diving deeper
05:00 GNR architecture and loss overview
09:50 Loss components
13:10 Diversity discriminator in more detail
16:25 Baseline comparisons
17:50 Ablations
18:40 Quantitative results (FID, DFID, LPIPS)
19:50 Controllability

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#gansandroses #anime #generativemodeling
 What's up? In this video I'm covering Gans and Roses stable, controllable, diverse image-to-image translation Works for videos too. I love the title. I mean, I think it's totally appropriate given that they are generating anime images So yeah, just roll with it by Minjin Chung and David Forsyth from the University of Illinois And let me just dive straight into what the paper is all about and it's about creating these cute images Basically anime images. So basically they feed in the content image here and they pick different styles for every single column Using this thing called style code vector and we'll soon see what it is and as you can see so they just pick out style code and they keep it the same across the whole column and One thing you can notice here is so first things first is that you can see that this hoodie here is preserved Because the content code is the same across this whole row Whereas the style code is changing and same thing here You can see this this formal color and it's being kept across the whole row because as I said the content code is fixed Whereas the style code is changing on the other hand if we just focus on a single column There we have that the content code is changing because we are having different input images here, right? but like the style code is kept the same and you can see that it's kind of interesting because You can see that the style is kind of kept although it's obviously very vague Wake concept and yeah, let me just show you one more thing and then I'll show you how this thing works. So This is what they get as a as a as a emerging phenomena basically from from their whole pipeline is that even though they don't train the the system on They don't have any temporal kind of a loss or constraint so and Despite that they can they can they can gain Really consistent frames so you can see here different input images So they'll they'll yield different content codes But the style code is kept the same and so you can see that basically it looks really Consistent like just going throughout the frames. You can see that the head pose here is the same as here But the style code so basically the style is the same so you can see this pointy beard Being the same there and here and that's really cool without any temporal constraints They achieve this if we compare their result with this baseline console again You can see that first things first so this looks terrible like the this is kind of frightening me and the second thing is that you can see that the Style is changing and and basically you can see here. The chain is kind of rounded here We have a pointy chin so it's not consistent across the frames So that basically means that the style is changing as well and it should have been kept fixed But it's this method is failing to do that on the other hand What they do here is they take this first frame and they take this last frame and this time So again, the content code is kind of determined by the input video and they are just kind of sampling randomly sampling different style codes and so you can see that This image here has a totally different appearance compared to this one. So they are totally two different animes So the style varies a lot here Even though they sample random random styles, they still get decently similar images So if you take a look at these two you can see that These two have almost the same face The only thing that's different is kind of the hair color and the background color But they are most of the same the eyes are the same so it's failing to create diverse set of anime avatars, so Those are the main points of this paper So they show that they can have really diverse set of avatars. They show that they can have this consistency and That's that's pretty much it. Let me now show you how how it works. I think the results are pretty impressive So let's yeah, let's start from from the beginning of the paper So so we show how to learn a map that takes a content code derived from from a face image and a randomly chosen Style code to an anime image we derive an adversarial loss from our simple and effective definitions of style and content And this adversarial loss guarantees the map is diverse a very wide range of anime can be produced from a single content code so they're stressing the diversity and basically, yeah So the most important step is to be exact about what is what is intended by content and what is intended by style So we adopt a specific definition content is what changes when face images are subject to a family of data augmentation Transformations and style is what does not change It's as easy as that and as you can see it's coupled to the definition of that set of augmentations and they say somewhere here so basically So know that this definition Is conditioned on the augmentations so different sets of data augmentations will result in different definitions of style So having said that let me just walk you through the pipeline. So let me explain how this works. So First things they do is they form a batch of input images Okay, and how they do that is they take a single image So maybe this one and they apply a bunch of different like geometrical type of augmentations like scaling rotation, etc. And So that's an important thing. I want you to understand here. So they don't take multiple different images. They take a single image They apply a bunch of different augmentations and then they feed that into this encoder So now what the trick here is is as you can see we have encoder We have decoder which kind of generates the anime images then we have additionally the second branch that again encodes this and decodes this and as you can probably assume already this is Like this will lead to cyclic consistency loss, but let me first focus on so there are three components to the loss function The first things first is that this style so these are the style codes coming from from these images And what they want to do is keep these the same So because they are they have the same image and they're just applying augmentations They want to make sure that the only thing that varies is the content code Whereas the style is the same because that's their definition of what content is and what style is so style is whatever remains the same And so this thing will encode that whatever remains the same concept and the content code will encode Whatever changes concept. Okay So how they will enforce that these are the same is they'll just find basically Variance of that because again imagine this S will be some I don't know like the dimensional vector and Because they have a batch of B images. They'll have B times the matrix and they'll just make sure that Basically the variance across different vectors Goes down to goes down to zero. So just formally written. That's variance of this batch of style codes should be one term that we are going to minimize and Obviously when variance is equal to zero all the vectors are the same The second part is so after they generate these images they'll feed them back here And as you can see so they want to preserve the content code So this C prime they want to make make sure that those are the same as these C's here So what they'll do is they'll feed the same style codes here and they'll take the content here and They'll try to reconstruct the very same images as in the input batch So they'll just do L to distance between the batch of real images and the batch That had some augmentations applied in between this batch of generated images and by doing that we we kind of So that the network will learn how to preserve the content code inside of the generated images The third component of the loss will be basically adversarial loss So that's your that's this part and this first branch is something you're familiar with Hopefully so that's just your your GAN so adversarial loss objective So we'll be training the discriminator to discriminate between a batch of real enemy images and a batch of generated images and on the other hand will be training generator to learn how to trick the discriminator into thinking that whatever it generated it comes from the real data set so that's our standard GAN objective But the interesting thing here is this mini batch as you can see mini batch standard deviation Let me try and zoom in here. So this is just standard deviation. So what that does is the following So if you just apply if you just have this branch here What will happen probably is that your generator can learn to basically collapse into a single? Image that looks realistic and discriminator will be totally fine with that I will just say that okay, this looks real to me, but we want to be diverse. We want to have diverse set of Avatars and so they what I do is they they take this these images and from from a certain layer here I think they are using penultimate layer We'll see the details a bit later, but they take the feature maps here and they just form these like basically standard deviation statistics and they'll use those to feed them into the second branch of discriminator and what that does is the following so that means that discriminator I the generator will need to learn how to mimic the basically those standard deviation across the whole batch and that will but by kind of trying to mimic the The same statistic from from the real images. You will learn how to create diverse set of images so that's the whole point of this thing now, let's dig into a bit more details and We'll see how all of those fit together Okay, so just ignore these I just verbal explained those so this was the first component So as you can see the loss has three components. That's the adversarial components. This is your classical again loss You have the style consistency loss and you have the cycle consistency loss. So the first thing is this as I said, so They say here in a batch as above we expect all Si sold the codes to be the same and this is how we enforce that concept and secondly, they have the cycle consistency loss and They say here that we find it helpful to shuffle the style codes in a batch. So there's just a minor detail Let me just show you the equations. So This part here is the encoder. So from y to x so that's that's this one So they are just kind of mixing different Notations here. They say here B to a they say y to x in the equation down there But that's the they want to see this thing so that's the equation represents you're feeding the generated image here and you out out comes the Content code and the style code and so what I say here is let me get back to the equation So this is the generated anime image and you feed back to get the content and the style code and then you feed it back So that very same content code, but you take the style code from from the original images So as you can see here, so those are So we take these style codes from the original Augmented images and we take whatever was generated here and we feed that into this decoder okay, so that's what these equations say and So a minor detail here is that as you can see so C comes from the Ith image but the style vector comes from the Jth image and What they did here by doing this is basically they are additionally encoding So this cycle consistency loss can also have the style consistency loss Mingled in and I don't know whether so this is obviously not that pretty Conceptually, but I guess that was some heuristic and they figured out they helps them further enforce that style consistency Component. Okay, and finally just L2. That's how we do cycle consistency loss So whatever we reconstructed we want to make sure that it's in the image space It's got a really small distance compared to the original augmented images There is one more detail here Additionally the they use this thing called LP IPS. So that's basically our perceptual loss from neural style transfer literature. So What happens is that let me go back to the diagram here. So what happens is that they Additionally, so they feed this image. So this is the input image and this is the reconstructed image they'll feed both of those through this generator and Decoder and basically they'll take certain feature maps from a certain layer and they'll want to make sure that also in the feature space They're really close looking at maybe again L2 metric So that's that's that additional part that they have in the cycle consistency component. Okay, and And finally, they have the diversity discriminator and adversarial loss component so That's the part I mentioned with the GAN loss and with this standard deviation stuff So let's see what they say our diversity discriminator exploits the mini badge standard deviation trick used in the progressive growing of GANs paper To exploit this difference in the penultimate layer of the discriminator We compute the standard deviation of each feature and pass that into another FC layers that's fully connected layer our diversity discriminator outputs the real fake logits and the standard deviation logits This means the discriminator can identify reliable differences in within batch variation so that's basically what I already explained and I'll just try to To tell you one thing that confuses me a little bit here. That's the following so imagine so this is your decoder and discriminator and basically What will happen once you feed this image? So let's say one image maybe this one once you feed it to and you extract the feature maps from this penultimate layer You'll basically have some activation volume. So those activations from this penultimate layer will look something like this So there'll be I will have some like let me draw it like this. So this will be some height This will be some width and I'll just do add a subscript here like L because we are looking at the layer L And we'll have some number of channels. So so basically these will be this is a collection of feature maps for this particular image, okay, so this will be C sub L and That's just a single image. So that's maybe this first image now We we do the same thing for for the second image. So we'll again have a similar type of activation volume there Etc and so the part that confuses me is that they haven't really explained how they are calculating these standard Deviation statistics. So what I assume is they take a certain feature like maybe this feature so that's basically, let's call it a I J and They're gonna compute the standard deviation across the channels and across the mini batch as well So now I don't know whether they are actually aggregating those standard deviations across the channels or whether they are just lumping that whole So where the final shape will be something like B times Height L times Width L or are they going to aggregate across this dimension and just feed that so it's kind of not clear The technicalities are not not kind of clear and I guess you have to experiment and try a couple things to see what works I think what I did in the progressive growing of GANs paper is that they Average across every single dimension and that then they just replicated into a certain constant feature map and then just append that to the Activation volumes, but yeah that part is not super clear. But the concept itself is hopefully Understandable and that's that they want to enforce That the generated images have the same similar or realistically looking statistics Which will make sure that we have diverse set of images. Okay so having said that let me continue here and Let's see the experiments and the results they got compared to the baselines Comparing this Gans and roses paper to these old baselines like grid++ console GAN and star GANv2 What we can notice is that if we take two different style codes here, we get completely different images So you can see that these two are okay Maybe the pose is similar, but the eyes are totally different. The chin is this is pointy chin This is kind of round chin and we fit and here the difference is even bigger So the head is obviously bigger the hair is different that the eyes are bigger etc So if we take a look at some other baselines here, you can see that it's pretty much the same picture Just the only thing that changes is kind of colors Same thing here and also like just take a look at this side Like this is so creepy and like there are some artifacts, but they look really similar So they and there are some artifacts like this one So they don't manage to create diverse types of avatars whereas this Gans and roses paper does they also compare with this newer? Any GAN paper and the same thing applies here as you can see applying three different style codes They they gain they get like totally different Like avatars so you can see there's there is a many differences between these three But taking a look at the end again output you can see that these three only like the only significant difference is maybe the eye The color basically and here you can also see the structure the face the eyes and everything changes So again a plus plus for the diversity component of the Gans and roses paper. They did some ablations Also like here you can see that this is the thing I was mentioning. So basically when you take this diversity discriminator Branch and you just just remove it you can see that that what happens is you get a basically mode collapse here and That means that all of these images are realistic and so the GAN component will say everything is fine But like as we can see they are pretty much the same and and they're yeah They're almost the same here when we include the branch. We actually get really diverse outputs and then they from some other paper they just tried this mode seeking loss and They begin just punch of artifacts. So they just stuck with this diversity discriminator component. Okay, they also did quantitative comparisons between with the baselines and just using these DFID and LPIPS Metrics which focus on the diversity of the images and they also use the FID so that's for shades inception distance We just focus is whether the generated image is actually Realistically looking and similar to the actual real avatars and Again, they just get basically much better Much better performance compared to the baseline. So DFID should be as low as possible You can see it's much lower than the baselines just focusing on this selfie to anime data set and FID is also smaller and LPIPS is higher. So all of these three metrics basically rely on a pre-trained classifier Like inception we three model or something and How they work is similar to the perceptual loss. So they just take they just do some Comparisons in the feature space and yeah, so I won't be digging into how they exactly work But main point here is that the quantitative results are are comparable and consistent with the qualitative results. We saw They also have There is this other paper that Created this technique called so this set of results Created this technique called so this sephapaper and they say that sephafinds laden direction that corresponds to large semantic changes in a style Based GAN by finding eigenvectors of the style modulation weight parameters We show that our style space is highly expressive and editable using these eigenvectors in figure seven So basically, they somehow calculate these eigenvectors and so every single image here was generated using a different style code And once you add these eigenvectors you can kind of Make them have certain properties. So if you add this eigenvector five You basically get small eyes and sharp features if you add eigenvector eight to get abstract lines And finally big eyes and dark hair if you add this eigenvector at 10 So just uh, they show that this manipulation method kind of works for for their style codes as well Okay, that's that's pretty much it I mean this paper is pretty pretty simple to understand and but the results are I like results Which a lot and so I thought covering it. So yeah, hopefully you found this video useful if you did consider sharing and subscribing And until next time. Bye. Bye
[{"start": 0.0, "end": 6.68, "text": " What's up? In this video I'm covering Gans and Roses stable, controllable, diverse image-to-image translation"}, {"start": 7.140000000000001, "end": 14.48, "text": " Works for videos too. I love the title. I mean, I think it's totally appropriate given that they are generating anime images"}, {"start": 14.48, "end": 19.92, "text": " So yeah, just roll with it by Minjin Chung and David Forsyth from the University of Illinois"}, {"start": 19.92, "end": 26.22, "text": " And let me just dive straight into what the paper is all about and it's about creating these cute images"}, {"start": 26.22, "end": 33.62, "text": " Basically anime images. So basically they feed in the content image here and they pick different styles for every single column"}, {"start": 33.96, "end": 37.34, "text": " Using this thing called style code vector and we'll soon see what it is"}, {"start": 37.34, "end": 42.94, "text": " and as you can see so they just pick out style code and they keep it the same across the whole column and"}, {"start": 43.9, "end": 50.739999999999995, "text": " One thing you can notice here is so first things first is that you can see that this hoodie here is preserved"}, {"start": 51.46, "end": 54.46, "text": " Because the content code is the same across this whole row"}, {"start": 54.46, "end": 58.7, "text": " Whereas the style code is changing and same thing here"}, {"start": 58.7, "end": 66.02, "text": " You can see this this formal color and it's being kept across the whole row because as I said the content code is fixed"}, {"start": 66.02, "end": 71.34, "text": " Whereas the style code is changing on the other hand if we just focus on a single column"}, {"start": 71.9, "end": 76.5, "text": " There we have that the content code is changing because we are having different input images here, right?"}, {"start": 76.58, "end": 80.46000000000001, "text": " but like the style code is kept the same and you can see that"}, {"start": 81.26, "end": 83.26, "text": " it's kind of interesting because"}, {"start": 83.26, "end": 89.86, "text": " You can see that the style is kind of kept although it's obviously very vague"}, {"start": 90.30000000000001, "end": 97.2, "text": " Wake concept and yeah, let me just show you one more thing and then I'll show you how this thing works. So"}, {"start": 98.10000000000001, "end": 101.5, "text": " This is what they get as a as a as a emerging phenomena"}, {"start": 101.5, "end": 106.78, "text": " basically from from their whole pipeline is that even though they don't train the the system on"}, {"start": 107.58000000000001, "end": 110.30000000000001, "text": " They don't have any temporal kind of a loss or constraint"}, {"start": 110.82000000000001, "end": 112.5, "text": " so and"}, {"start": 112.5, "end": 114.86, "text": " Despite that they can they can they can gain"}, {"start": 115.58, "end": 120.18, "text": " Really consistent frames so you can see here different input images"}, {"start": 120.18, "end": 123.1, "text": " So they'll they'll yield different content codes"}, {"start": 123.14, "end": 127.82, "text": " But the style code is kept the same and so you can see that basically it looks really"}, {"start": 128.26, "end": 133.14, "text": " Consistent like just going throughout the frames. You can see that the head pose here is the same as here"}, {"start": 133.14, "end": 137.7, "text": " But the style code so basically the style is the same so you can see this pointy beard"}, {"start": 137.7, "end": 143.22, "text": " Being the same there and here and that's really cool without any temporal constraints"}, {"start": 143.22, "end": 146.66, "text": " They achieve this if we compare their result with this"}, {"start": 147.33999999999997, "end": 148.78, "text": " baseline console again"}, {"start": 148.78, "end": 150.5, "text": " You can see that first things first"}, {"start": 150.5, "end": 156.66, "text": " so this looks terrible like the this is kind of frightening me and the second thing is that you can see that the"}, {"start": 157.06, "end": 161.54, "text": " Style is changing and and basically you can see here. The chain is kind of rounded here"}, {"start": 161.54, "end": 165.0, "text": " We have a pointy chin so it's not consistent across the frames"}, {"start": 165.0, "end": 170.02, "text": " So that basically means that the style is changing as well and it should have been kept fixed"}, {"start": 170.02, "end": 173.54, "text": " But it's this method is failing to do that on the other hand"}, {"start": 173.54, "end": 179.38, "text": " What they do here is they take this first frame and they take this last frame and this time"}, {"start": 180.54, "end": 187.98, "text": " So again, the content code is kind of determined by the input video and they are just kind of sampling randomly sampling different"}, {"start": 188.06, "end": 190.06, "text": " style codes and so you can see that"}, {"start": 190.06, "end": 195.84, "text": " This image here has a totally different appearance compared to this one. So they are totally two different animes"}, {"start": 196.14000000000001, "end": 198.66, "text": " So the style varies a lot here"}, {"start": 199.18, "end": 204.46, "text": " Even though they sample random random styles, they still get decently similar images"}, {"start": 204.46, "end": 207.12, "text": " So if you take a look at these two you can see that"}, {"start": 207.54, "end": 209.5, "text": " These two have almost the same face"}, {"start": 209.5, "end": 213.54, "text": " The only thing that's different is kind of the hair color and the background color"}, {"start": 213.54, "end": 218.34, "text": " But they are most of the same the eyes are the same so it's failing to create diverse set of"}, {"start": 218.34, "end": 221.02, "text": " anime avatars, so"}, {"start": 221.78, "end": 223.82, "text": " Those are the main points of this paper"}, {"start": 223.82, "end": 230.78, "text": " So they show that they can have really diverse set of avatars. They show that they can have this consistency and"}, {"start": 232.02, "end": 236.98000000000002, "text": " That's that's pretty much it. Let me now show you how how it works. I think the results are pretty impressive"}, {"start": 237.38, "end": 239.78, "text": " So let's yeah, let's start from from the beginning of the paper"}, {"start": 239.78, "end": 245.7, "text": " So so we show how to learn a map that takes a content code derived from from a face image and a randomly chosen"}, {"start": 245.7, "end": 252.66, "text": " Style code to an anime image we derive an adversarial loss from our simple and effective definitions of style and content"}, {"start": 252.66, "end": 260.14, "text": " And this adversarial loss guarantees the map is diverse a very wide range of anime can be produced from a single content code"}, {"start": 260.14, "end": 262.14, "text": " so they're stressing the diversity and"}, {"start": 262.98, "end": 264.82, "text": " basically, yeah"}, {"start": 264.82, "end": 270.76, "text": " So the most important step is to be exact about what is what is intended by content and what is intended by style"}, {"start": 270.76, "end": 276.94, "text": " So we adopt a specific definition content is what changes when face images are subject to a family of data augmentation"}, {"start": 277.62, "end": 280.26, "text": " Transformations and style is what does not change"}, {"start": 280.34, "end": 286.5, "text": " It's as easy as that and as you can see it's coupled to the definition of that set of augmentations and they say somewhere here"}, {"start": 286.5, "end": 288.26, "text": " so basically"}, {"start": 288.26, "end": 290.26, "text": " So know that this definition"}, {"start": 290.65999999999997, "end": 297.38, "text": " Is conditioned on the augmentations so different sets of data augmentations will result in different definitions of style"}, {"start": 297.38, "end": 303.58, "text": " So having said that let me just walk you through the pipeline. So let me explain how this works. So"}, {"start": 304.58, "end": 307.65999999999997, "text": " First things they do is they form a batch of input images"}, {"start": 307.65999999999997, "end": 310.42, "text": " Okay, and how they do that is they take a single image"}, {"start": 310.42, "end": 314.62, "text": " So maybe this one and they apply a bunch of different like"}, {"start": 315.26, "end": 319.38, "text": " geometrical type of augmentations like scaling rotation, etc. And"}, {"start": 320.1, "end": 323.86, "text": " So that's an important thing. I want you to understand here. So they don't take"}, {"start": 323.86, "end": 327.18, "text": " multiple different images. They take a single image"}, {"start": 327.18, "end": 331.18, "text": " They apply a bunch of different augmentations and then they feed that into this encoder"}, {"start": 331.3, "end": 336.46000000000004, "text": " So now what the trick here is is as you can see we have encoder"}, {"start": 336.46000000000004, "end": 342.66, "text": " We have decoder which kind of generates the anime images then we have additionally the second branch"}, {"start": 343.06, "end": 349.14, "text": " that again encodes this and decodes this and as you can probably assume already this is"}, {"start": 349.14, "end": 356.06, "text": " Like this will lead to cyclic consistency loss, but let me first focus on so there are three components to the loss function"}, {"start": 356.06, "end": 362.18, "text": " The first things first is that this style so these are the style codes coming from from these images"}, {"start": 362.18, "end": 366.34, "text": " And what they want to do is keep these the same"}, {"start": 366.38, "end": 370.47999999999996, "text": " So because they are they have the same image and they're just applying augmentations"}, {"start": 370.5, "end": 374.21999999999997, "text": " They want to make sure that the only thing that varies is the content code"}, {"start": 374.22, "end": 381.34000000000003, "text": " Whereas the style is the same because that's their definition of what content is and what style is so style is whatever remains the same"}, {"start": 381.34000000000003, "end": 388.66, "text": " And so this thing will encode that whatever remains the same concept and the content code will encode"}, {"start": 388.66, "end": 390.66, "text": " Whatever changes concept. Okay"}, {"start": 390.90000000000003, "end": 395.64000000000004, "text": " So how they will enforce that these are the same is they'll just find basically"}, {"start": 397.06, "end": 403.42, "text": " Variance of that because again imagine this S will be some I don't know like"}, {"start": 403.42, "end": 405.42, "text": " the dimensional vector and"}, {"start": 405.82, "end": 412.44, "text": " Because they have a batch of B images. They'll have B times the matrix and they'll just make sure that"}, {"start": 413.66, "end": 417.06, "text": " Basically the variance across different vectors"}, {"start": 417.62, "end": 423.54, "text": " Goes down to goes down to zero. So just formally written. That's variance of this"}, {"start": 424.1, "end": 431.26, "text": " batch of style codes should be one term that we are going to minimize and"}, {"start": 431.26, "end": 435.38, "text": " Obviously when variance is equal to zero all the vectors are the same"}, {"start": 435.5, "end": 440.3, "text": " The second part is so after they generate these images they'll feed them back here"}, {"start": 440.3, "end": 443.18, "text": " And as you can see so they want to preserve the content code"}, {"start": 443.18, "end": 448.18, "text": " So this C prime they want to make make sure that those are the same as these C's here"}, {"start": 448.18, "end": 455.09999999999997, "text": " So what they'll do is they'll feed the same style codes here and they'll take the content here and"}, {"start": 455.21999999999997, "end": 458.88, "text": " They'll try to reconstruct the very same images as in the input batch"}, {"start": 458.88, "end": 464.58, "text": " So they'll just do L to distance between the batch of real images and the batch"}, {"start": 464.82, "end": 471.24, "text": " That had some augmentations applied in between this batch of generated images and by doing that we we kind of"}, {"start": 471.64, "end": 477.28, "text": " So that the network will learn how to preserve the content code inside of the generated images"}, {"start": 478.12, "end": 482.04, "text": " The third component of the loss will be basically adversarial loss"}, {"start": 482.15999999999997, "end": 487.28, "text": " So that's your that's this part and this first branch is something you're familiar with"}, {"start": 487.28, "end": 492.71999999999997, "text": " Hopefully so that's just your your GAN so adversarial loss objective"}, {"start": 492.71999999999997, "end": 499.02, "text": " So we'll be training the discriminator to discriminate between a batch of real enemy images and a batch of"}, {"start": 499.44, "end": 503.32, "text": " generated images and on the other hand will be training generator to learn how to"}, {"start": 503.88, "end": 505.52, "text": " trick the"}, {"start": 505.52, "end": 511.84, "text": " discriminator into thinking that whatever it generated it comes from the real data set so that's our standard GAN objective"}, {"start": 511.84, "end": 516.28, "text": " But the interesting thing here is this mini batch as you can see mini batch standard deviation"}, {"start": 516.28, "end": 520.72, "text": " Let me try and zoom in here. So this is just standard deviation. So what that does is the following"}, {"start": 520.72, "end": 523.0799999999999, "text": " So if you just apply if you just have this branch here"}, {"start": 523.12, "end": 530.9599999999999, "text": " What will happen probably is that your generator can learn to basically collapse into a single?"}, {"start": 531.3199999999999, "end": 535.24, "text": " Image that looks realistic and discriminator will be totally fine with that"}, {"start": 535.24, "end": 541.3199999999999, "text": " I will just say that okay, this looks real to me, but we want to be diverse. We want to have diverse set of"}, {"start": 541.32, "end": 549.48, "text": " Avatars and so they what I do is they they take this these images and from from a certain layer here"}, {"start": 549.48, "end": 552.08, "text": " I think they are using penultimate layer"}, {"start": 552.08, "end": 557.1600000000001, "text": " We'll see the details a bit later, but they take the feature maps here and they just form these"}, {"start": 558.0, "end": 565.96, "text": " like basically standard deviation statistics and they'll use those to feed them into the second branch of discriminator and what that does is"}, {"start": 565.96, "end": 567.96, "text": " the following so that means that"}, {"start": 567.96, "end": 572.1600000000001, "text": " discriminator I the generator will need to learn how to mimic the"}, {"start": 573.12, "end": 576.6800000000001, "text": " basically those standard deviation across the whole batch and that will"}, {"start": 577.36, "end": 580.2, "text": " but by kind of trying to mimic the"}, {"start": 580.9200000000001, "end": 585.6, "text": " The same statistic from from the real images. You will learn how to create diverse set of images"}, {"start": 585.6, "end": 590.12, "text": " so that's the whole point of this thing now, let's dig into a bit more details and"}, {"start": 591.4000000000001, "end": 593.9200000000001, "text": " We'll see how all of those fit together"}, {"start": 593.92, "end": 599.92, "text": " Okay, so just ignore these I just verbal explained those so this was the first component"}, {"start": 599.92, "end": 605.4399999999999, "text": " So as you can see the loss has three components. That's the adversarial components. This is your classical again loss"}, {"start": 605.4399999999999, "end": 612.68, "text": " You have the style consistency loss and you have the cycle consistency loss. So the first thing is this as I said, so"}, {"start": 613.4799999999999, "end": 616.8, "text": " They say here in a batch as above we expect all"}, {"start": 617.24, "end": 621.24, "text": " Si sold the codes to be the same and this is how we enforce"}, {"start": 621.24, "end": 623.5600000000001, "text": " that concept and"}, {"start": 624.84, "end": 627.64, "text": " secondly, they have the cycle consistency loss and"}, {"start": 628.5600000000001, "end": 634.12, "text": " They say here that we find it helpful to shuffle the style codes in a batch. So there's just a minor detail"}, {"start": 635.08, "end": 637.08, "text": " Let me just show you the equations. So"}, {"start": 637.5600000000001, "end": 643.08, "text": " This part here is the encoder. So from y to x so that's that's this one"}, {"start": 643.84, "end": 645.84, "text": " So they are just kind of mixing different"}, {"start": 646.4, "end": 651.0, "text": " Notations here. They say here B to a they say y to x in the equation down there"}, {"start": 651.0, "end": 653.32, "text": " But that's the they want to see this thing"}, {"start": 653.32, "end": 660.22, "text": " so that's the equation represents you're feeding the generated image here and you out out comes the"}, {"start": 660.6, "end": 666.26, "text": " Content code and the style code and so what I say here is let me get back to the equation"}, {"start": 666.88, "end": 673.36, "text": " So this is the generated anime image and you feed back to get the content and the style code and then you feed it back"}, {"start": 674.16, "end": 679.36, "text": " So that very same content code, but you take the style code from from the original images"}, {"start": 679.36, "end": 682.2, "text": " So as you can see here, so those are"}, {"start": 683.32, "end": 686.64, "text": " So we take these style codes from the original"}, {"start": 687.24, "end": 692.52, "text": " Augmented images and we take whatever was generated here and we feed that into this decoder"}, {"start": 692.52, "end": 694.84, "text": " okay, so that's what these equations say and"}, {"start": 695.76, "end": 700.9, "text": " So a minor detail here is that as you can see so C comes from the Ith image"}, {"start": 701.12, "end": 704.16, "text": " but the style vector comes from the Jth image and"}, {"start": 704.16, "end": 708.92, "text": " What they did here by doing this is basically they are additionally encoding"}, {"start": 708.92, "end": 713.4599999999999, "text": " So this cycle consistency loss can also have the style consistency loss"}, {"start": 714.0, "end": 717.86, "text": " Mingled in and I don't know whether so this is obviously not that pretty"}, {"start": 718.28, "end": 722.8, "text": " Conceptually, but I guess that was some heuristic and they figured out they helps them further"}, {"start": 723.3199999999999, "end": 725.3199999999999, "text": " enforce that style consistency"}, {"start": 726.12, "end": 731.54, "text": " Component. Okay, and finally just L2. That's how we do cycle consistency loss"}, {"start": 731.54, "end": 735.48, "text": " So whatever we reconstructed we want to make sure that it's in the image space"}, {"start": 735.48, "end": 739.04, "text": " It's got a really small distance compared to the original"}, {"start": 739.88, "end": 741.56, "text": " augmented images"}, {"start": 741.56, "end": 743.56, "text": " There is one more detail here"}, {"start": 744.28, "end": 751.04, "text": " Additionally the they use this thing called LP IPS. So that's basically our perceptual loss from neural style transfer literature. So"}, {"start": 751.8, "end": 757.04, "text": " What happens is that let me go back to the diagram here. So what happens is that they"}, {"start": 757.04, "end": 763.04, "text": " Additionally, so they feed this image. So this is the input image and this is the reconstructed image"}, {"start": 763.1999999999999, "end": 766.52, "text": " they'll feed both of those through this generator and"}, {"start": 767.9599999999999, "end": 775.04, "text": " Decoder and basically they'll take certain feature maps from a certain layer and they'll want to make sure that also in the feature space"}, {"start": 775.16, "end": 778.3199999999999, "text": " They're really close looking at maybe again L2 metric"}, {"start": 778.3199999999999, "end": 784.24, "text": " So that's that's that additional part that they have in the cycle consistency component. Okay, and"}, {"start": 784.24, "end": 789.04, "text": " And finally, they have the diversity discriminator and adversarial loss"}, {"start": 789.52, "end": 791.0, "text": " component so"}, {"start": 791.0, "end": 795.48, "text": " That's the part I mentioned with the GAN loss and with this standard deviation stuff"}, {"start": 795.48, "end": 802.92, "text": " So let's see what they say our diversity discriminator exploits the mini badge standard deviation trick used in the progressive growing of GANs paper"}, {"start": 803.2, "end": 806.92, "text": " To exploit this difference in the penultimate layer of the discriminator"}, {"start": 806.92, "end": 811.08, "text": " We compute the standard deviation of each feature and pass that into another"}, {"start": 811.08, "end": 818.1600000000001, "text": " FC layers that's fully connected layer our diversity discriminator outputs the real fake logits and the standard deviation logits"}, {"start": 818.1600000000001, "end": 823.48, "text": " This means the discriminator can identify reliable differences in within batch variation"}, {"start": 823.6800000000001, "end": 827.76, "text": " so that's basically what I already explained and I'll just try to"}, {"start": 828.8000000000001, "end": 834.72, "text": " To tell you one thing that confuses me a little bit here. That's the following so imagine so this is your"}, {"start": 835.24, "end": 837.24, "text": " decoder and discriminator and"}, {"start": 838.1600000000001, "end": 839.36, "text": " basically"}, {"start": 839.36, "end": 841.72, "text": " What will happen once you feed this image?"}, {"start": 841.72, "end": 847.32, "text": " So let's say one image maybe this one once you feed it to and you extract the feature maps from this penultimate layer"}, {"start": 847.32, "end": 852.6, "text": " You'll basically have some activation volume. So those activations from this penultimate layer will look something like this"}, {"start": 852.6, "end": 858.8000000000001, "text": " So there'll be I will have some like let me draw it like this. So this will be some height"}, {"start": 859.76, "end": 866.44, "text": " This will be some width and I'll just do add a subscript here like L because we are looking at the layer L"}, {"start": 866.44, "end": 872.2600000000001, "text": " And we'll have some number of channels. So so basically these will be this is a collection of feature maps"}, {"start": 872.9200000000001, "end": 876.24, "text": " for this particular image, okay, so this will be C sub L and"}, {"start": 877.4000000000001, "end": 880.6800000000001, "text": " That's just a single image. So that's maybe this first image now"}, {"start": 880.6800000000001, "end": 887.5200000000001, "text": " We we do the same thing for for the second image. So we'll again have a similar type of activation volume"}, {"start": 888.24, "end": 889.4000000000001, "text": " there"}, {"start": 889.4000000000001, "end": 895.8000000000001, "text": " Etc and so the part that confuses me is that they haven't really explained how they are calculating these standard"}, {"start": 895.8, "end": 900.3199999999999, "text": " Deviation statistics. So what I assume is they take a certain feature like maybe this feature"}, {"start": 900.88, "end": 902.4399999999999, "text": " so that's"}, {"start": 902.4399999999999, "end": 904.68, "text": " basically, let's call it a I"}, {"start": 905.8, "end": 907.3199999999999, "text": " J and"}, {"start": 907.3199999999999, "end": 913.64, "text": " They're gonna compute the standard deviation across the channels and across the mini batch as well"}, {"start": 913.7199999999999, "end": 921.4799999999999, "text": " So now I don't know whether they are actually aggregating those standard deviations across the channels or whether they are just lumping that whole"}, {"start": 921.48, "end": 925.72, "text": " So where the final shape will be something like B"}, {"start": 926.48, "end": 928.48, "text": " times"}, {"start": 929.2, "end": 930.72, "text": " Height L"}, {"start": 930.72, "end": 931.8000000000001, "text": " times"}, {"start": 931.8000000000001, "end": 938.7, "text": " Width L or are they going to aggregate across this dimension and just feed that so it's kind of not clear"}, {"start": 938.7, "end": 945.52, "text": " The technicalities are not not kind of clear and I guess you have to experiment and try a couple things to see what works"}, {"start": 945.88, "end": 949.96, "text": " I think what I did in the progressive growing of GANs paper is that they"}, {"start": 949.96, "end": 955.9200000000001, "text": " Average across every single dimension and that then they just replicated into a certain constant feature map"}, {"start": 955.9200000000001, "end": 958.0, "text": " and then just append that to the"}, {"start": 958.9200000000001, "end": 964.0, "text": " Activation volumes, but yeah that part is not super clear. But the concept itself is hopefully"}, {"start": 965.0, "end": 967.0, "text": " Understandable and that's that they want to enforce"}, {"start": 967.2800000000001, "end": 973.0400000000001, "text": " That the generated images have the same similar or realistically looking statistics"}, {"start": 973.36, "end": 976.32, "text": " Which will make sure that we have diverse set of images. Okay"}, {"start": 976.32, "end": 980.5200000000001, "text": " so having said that let me continue here and"}, {"start": 981.6800000000001, "end": 984.96, "text": " Let's see the experiments and the results they got compared to the baselines"}, {"start": 986.32, "end": 988.32, "text": " Comparing this"}, {"start": 988.32, "end": 993.2, "text": " Gans and roses paper to these old baselines like grid++ console GAN and star GANv2"}, {"start": 993.2, "end": 998.5400000000001, "text": " What we can notice is that if we take two different style codes here, we get completely different images"}, {"start": 998.5400000000001, "end": 1001.5200000000001, "text": " So you can see that these two are okay"}, {"start": 1001.52, "end": 1006.38, "text": " Maybe the pose is similar, but the eyes are totally different. The chin is this is pointy chin"}, {"start": 1006.38, "end": 1010.64, "text": " This is kind of round chin and we fit and here the difference is even bigger"}, {"start": 1010.64, "end": 1014.84, "text": " So the head is obviously bigger the hair is different that the eyes are bigger etc"}, {"start": 1014.84, "end": 1019.12, "text": " So if we take a look at some other baselines here, you can see that it's pretty much the same picture"}, {"start": 1019.12, "end": 1021.36, "text": " Just the only thing that changes is kind of colors"}, {"start": 1021.86, "end": 1024.92, "text": " Same thing here and also like just take a look at this side"}, {"start": 1024.92, "end": 1029.96, "text": " Like this is so creepy and like there are some artifacts, but they look really similar"}, {"start": 1029.96, "end": 1032.64, "text": " So they and there are some artifacts like this one"}, {"start": 1032.64, "end": 1040.1200000000001, "text": " So they don't manage to create diverse types of avatars whereas this Gans and roses paper does they also compare with this newer?"}, {"start": 1040.48, "end": 1046.3600000000001, "text": " Any GAN paper and the same thing applies here as you can see applying three different style codes"}, {"start": 1046.48, "end": 1049.48, "text": " They they gain they get like totally different"}, {"start": 1049.8400000000001, "end": 1054.88, "text": " Like avatars so you can see there's there is a many differences between these three"}, {"start": 1054.88, "end": 1061.96, "text": " But taking a look at the end again output you can see that these three only like the only significant difference is maybe the eye"}, {"start": 1061.96, "end": 1066.72, "text": " The color basically and here you can also see the structure the face the eyes and everything changes"}, {"start": 1066.72, "end": 1073.0400000000002, "text": " So again a plus plus for the diversity component of the Gans and roses paper. They did some ablations"}, {"start": 1073.5200000000002, "end": 1080.88, "text": " Also like here you can see that this is the thing I was mentioning. So basically when you take this diversity"}, {"start": 1082.0800000000002, "end": 1083.3600000000001, "text": " discriminator"}, {"start": 1083.36, "end": 1090.7199999999998, "text": " Branch and you just just remove it you can see that that what happens is you get a basically mode collapse here and"}, {"start": 1091.84, "end": 1097.04, "text": " That means that all of these images are realistic and so the GAN component will say everything is fine"}, {"start": 1097.12, "end": 1101.1599999999999, "text": " But like as we can see they are pretty much the same and and they're yeah"}, {"start": 1101.1599999999999, "end": 1106.24, "text": " They're almost the same here when we include the branch. We actually get really diverse outputs"}, {"start": 1106.6399999999999, "end": 1110.9599999999998, "text": " and then they from some other paper they just tried this mode seeking loss and"}, {"start": 1110.96, "end": 1118.96, "text": " They begin just punch of artifacts. So they just stuck with this diversity discriminator component. Okay, they also did"}, {"start": 1120.16, "end": 1124.48, "text": " quantitative comparisons between with the baselines and just using these"}, {"start": 1125.28, "end": 1127.28, "text": " DFID and LPIPS"}, {"start": 1128.72, "end": 1134.28, "text": " Metrics which focus on the diversity of the images and they also use the FID so that's for shades inception distance"}, {"start": 1134.28, "end": 1137.1200000000001, "text": " We just focus is whether the generated image is actually"}, {"start": 1137.12, "end": 1142.2399999999998, "text": " Realistically looking and similar to the actual real avatars and"}, {"start": 1143.04, "end": 1146.2399999999998, "text": " Again, they just get basically much better"}, {"start": 1147.1999999999998, "end": 1151.6, "text": " Much better performance compared to the baseline. So DFID should be as low as possible"}, {"start": 1151.6, "end": 1157.9199999999998, "text": " You can see it's much lower than the baselines just focusing on this selfie to anime data set and FID is also smaller and"}, {"start": 1158.4799999999998, "end": 1164.3999999999999, "text": " LPIPS is higher. So all of these three metrics basically rely on a pre-trained classifier"}, {"start": 1164.4, "end": 1167.44, "text": " Like inception we three model or something and"}, {"start": 1168.0800000000002, "end": 1172.8000000000002, "text": " How they work is similar to the perceptual loss. So they just take they just do some"}, {"start": 1173.76, "end": 1178.4, "text": " Comparisons in the feature space and yeah, so I won't be digging into how they exactly work"}, {"start": 1178.4, "end": 1184.88, "text": " But main point here is that the quantitative results are are comparable and consistent with the qualitative results. We saw"}, {"start": 1186.0, "end": 1187.52, "text": " They also have"}, {"start": 1187.52, "end": 1189.76, "text": " There is this other paper that"}, {"start": 1190.3200000000002, "end": 1192.72, "text": " Created this technique called so this set of results"}, {"start": 1192.72, "end": 1200.72, "text": " Created this technique called so this sephapaper and they say that sephafinds laden direction that corresponds to large semantic changes in a style"}, {"start": 1200.72, "end": 1204.8, "text": " Based GAN by finding eigenvectors of the style modulation weight parameters"}, {"start": 1204.96, "end": 1210.72, "text": " We show that our style space is highly expressive and editable using these eigenvectors in figure seven"}, {"start": 1210.96, "end": 1218.56, "text": " So basically, they somehow calculate these eigenvectors and so every single image here was generated using a different style code"}, {"start": 1218.8, "end": 1221.44, "text": " And once you add these eigenvectors you can kind of"}, {"start": 1221.44, "end": 1226.0, "text": " Make them have certain properties. So if you add this eigenvector five"}, {"start": 1226.0800000000002, "end": 1231.28, "text": " You basically get small eyes and sharp features if you add eigenvector eight to get abstract lines"}, {"start": 1231.6000000000001, "end": 1235.04, "text": " And finally big eyes and dark hair if you add this eigenvector at 10"}, {"start": 1235.3600000000001, "end": 1241.68, "text": " So just uh, they show that this manipulation method kind of works for for their style codes as well"}, {"start": 1242.0800000000002, "end": 1243.68, "text": " Okay, that's that's pretty much it"}, {"start": 1243.68, "end": 1248.56, "text": " I mean this paper is pretty pretty simple to understand and but the results are I like results"}, {"start": 1248.56, "end": 1256.58, "text": " Which a lot and so I thought covering it. So yeah, hopefully you found this video useful if you did consider sharing and subscribing"}, {"start": 1256.58, "end": 1278.58, "text": " And until next time. Bye. Bye"}]