Text Style Brush - Transfer of text aesthetics from a single example | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I explain the "Text Style Brush - Transfer of text aesthetics from a single example" paper from Facebook AI research.

They achieve a high-quality, single-shot, text style transfer across multiple domains (handwritten, scene text) through a combination of self-supervised and supervised learning.

This work promises many exciting future applications, like augmented/mixed reality, but also raises societal concerns related to deep fakes.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://ai.facebook.com/research/publications/textstylebrush-transfer-of-text-aesthetics-from-a-single-example/
✅ Blog: https://ai.facebook.com/blog/ai-can-now-emulate-text-style-in-images-in-one-shot-using-just-a-single-word
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 What is Text Style Brush about?
02:35 Key points of the paper
04:05 A high-level overview of the TSB pipeline
13:00 Diving deeper - Encoders
14:30 Diving deeper - Generator and map network
15:30 Adaptive Instance Normalization
17:40 Text perceptual loss
21:15 Text content loss
23:10 Reconstruction loss
27:20 New Imgur 5k dataset
28:05 Ablations
30:15 Text reco accuracy, quantitative quality, user study
32:40 Failure cases
33:40 Future applications and deep fakes
35:10 Conclusions

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#textstyletransfer #deepfakes #facebookai
 What's up? In this video I'm covering this new paper called Textile Brush Transfer of Text Aesthetics from a Single Example by Pravin Krishnan, Ramakavuri, Guampeng, Boris Vasilev and Tal Hessner from Facebook AI Research. Basically they got some really nice results of rendering text in a specified style, in a one-shot setting. And let me show you what I mean by that. So basically you can see here in the source image you have some text and they kind of find a bounty box around it and they extract it and you can see just using this single word image they can learn the style and render novel content in that very same style one-shot. So basically just a single image like this one and you can do something like this. So they just kind of swapped a soya bottle for a tea vessel. And not only that, so you can see here another example on the book and not only that but they can also work cross-domain. So they're not constrained to a scene text editing domain. They can also do handwriting. So again here you can see a single word image and they can render novel sentences using that very same style. So the thing is they only generate these word images and how they actually blend these together is using something called Poisson blending. That's the only part that feels a bit like hand crafting compared to this deep learning approach of actually generating these images. So let me show you some more examples what they achieve and then we'll start explaining the paper. So here you can see some examples on the left you have word image and on the right you can see the novel content in the same style. And it's pretty nice. There are some problems obviously here for example background is not the same. Like words are kind of blurry and I'll later give my hypothesis why that is and I think it's because of the reconstruction loss they're using with L1. But it's still pretty decent. So again here you can see the foreground texture is kind of lost here. The letters are kind of blurred. But all in all like it's really nice, looks awesome. Also sometimes I notice that the perspective is not the same so they don't manage to capture the sheer, the perspective of the source style. But like it's pretty awesome. Like look at this one. Disarming statement. I wouldn't be able to kind of discriminate between the two. They look the same to me. So nice results. Now let me start digging into how it works. So they say here we present a novel approach for disentangling the content of a text image from all aspects of its appearance. The appearance representation we derive can then be applied to new content for one shot transfer of the source style to new content. We'll learn this disentanglement in a self-supervised manner. So this is not completely true because they actually have a couple of pre-training networks like one for, like one is an OCR engine, one is a font classifier and they also assume to have those bounding boxes as well as the ground truth text inside of those bounding boxes. So there, it's a combination of supervised learning and self-supervised learning, but like a nice portion of the pipeline is actually self-supervised. So that's cool. So our method processes entire word boxes without requiring a segmentation of text from background per character processing or making assumptions on string lengths. So that's cool. So as a kind of motivation for one shot learning, they say it here, moreover, it's unreasonable to expect text images used as style samples to provide example appearances for the entire alphabet, including digits. Transferring styles between strings with different characters therefore implies hallucinating plausible appearances of unseen characters in the input style. And that kind of, yeah, should associate you with generative adversarial networks and that's one of the components they're obviously using here in this work, but there are many components to it. And let me now kind of try and explain. So I'll first try to give you like a high level overview and then we'll start digging into every single component. And like most of this video, we'll focus on explaining how this thing works because the problem itself is ill-posed as every, like similarly to neural style transfer because as a final conclusion, like humans just make a decision whether something looks nice or not. Or although there are some objective metrics like Fraschay's inception distance or inception score used in GANs, but like those are not that robust. And yeah, what they did also was a user study. So that's additionally, they've showed that this work looks better for a certain group of people they took for this experiment. So let me now start kind of dissecting this. So first things first, what they do is they take the image that has the style, like the style they want, and they crop it around the image word. So they put some context around the actual text that has the style they want. And that's one input. The second thing they do is they have a text which they want to render in the style from this image. And so you can see here, so earth is the, so they just take the word earth, which is the same as here, because as I told you, they have the ground truth, but they add some novel text for which they don't have the actual image. And what they then do is they don't input the text itself. They actually render the text on a white background and using some standard font. And we'll see those details a bit later. So once they render the text, now they can encode the text using this content encoder and they can encode the style using this style encoder. Both of these are just simple ResNet, basically ResNet, 34. And it's just modified a little bit. And so what I do here is because they have, as I told you, this bounding box ground truth, what I do is they take specific features corresponding to the bounding box and they do an average pooling along those pixels, along those features, and they create this vector here, which is a style vector, which was like, I think, 512, it doesn't matter, but yeah, let's make it a bit concrete. The thing they do with content is they just create a tensor here. Like as I said, they modify the ResNet 34, so they take just a certain feature maps from a certain layer. And the next thing they do is, you can see here, they feed those into this generator, which is style again, modified generator pretty much. So this style mapping network, what it does is the following. So it maps this style representation, this latent vector here, into multiple coefficients. And what these do if you're familiar with either the original adaptive instance normalization paper or with style again, is they kind of modify the feature maps, like they modify certain statistics like the mean and standard deviation so that they can kind of build in the style of their choosing, like the style they want. Okay? So I'll later explain how that adaptive instance normalization works, but for now you just have some coefficients here, which we got by mapping it from this style vector, and they modified, they use those to modify the feature maps. Kind of going, zooming in a little bit here, you can see the architecture is quite detailed. There are many components here, but the main idea is to use this modulated convolutions, so to use these coefficients to kind of modify the feature maps, and then they just have a progressive growing of these feature maps until they generate as the output this word image as well as the soft mask. So they're generating those two, so both the soft masks as well as the actual image words. So, okay. So once they generate those, they have five losses in total. So as you can see, this is basically hand crafting 2.0. They're using just DL components to create these systems, but the thing I don't like is that there are so many moving parts here, and I hope that like a couple of papers down the line, this will be something like end to end and we'll do the same thing. Okay, rent over. Let me explain to you briefly these losses functions. So they have, so these two here, so the typeface classifier and the recognizer are pre-trained and frozen during the training. So what I do is they take a, so this is a VGG 19 network, so this one is VGG 19. What I do is they take a VGG 19 network and they have some synthetic data set with certain image words, and they have actually the ground truth labels for the font, for the font type. So they just train a simple classifier that can predict what's the font of the input image. And a thing worth mentioning here is that the fonts used in that data set won't be the same as the fonts that this model will eventually be using because like they're using real data and here they just have synthetic fonts. So yeah, you can't cover the whole variance like that's present in real data sets. So having said that, what I do here is something, so this is a perceptual loss. If you're familiar with neural cell transfer, what they do here is they basically make sure, so as you can see, they input the ground truth, so this is this image, red image, and they input this generated image. And what I want to make sure is that the feature maps, it's a specific layer, are the same looking at one distance, for example. And also they form gram matrices which kind of capture the style of the image and they also make sure that those look the same for both this generated image as well as for this ground truth image. They also have some embedding loss, we'll get to that a bit later, but that's pretty much a neural style transfer part. Then they have recognizer which is a simple OCR engine. Again, they trained it, they took some pre-trained OCR engine, off the shelf thing, and what they do is because we have the ground truth text, we know that this is earth and warming, they just input all of these four, so both the masks as well as the generated images, and they kind of do the OCR. And finally they do simple cross entropy between the ground truth text and between whatever this OCR engine saw in these four images. So yeah, basically, let me just kind of explain that a bit more detailed. So let's say the first character, so it's out with certain distribution and you're gonna do argmax and that will be our first character. And so what they do is because they have the ground truth words, for example, this is the, let's say we're looking at this image, so they know that the first letter is E, so that will be one hot encoding of E, and they'll just do cross entropy between the output distribution and between the target distribution to train the generator. So again, this is frozen, so they are not modifying these weights, they are modifying the generator and the rest of the pipeline, okay? Same thing for this network here. And finally they have the, like this is pretty much default thing, discriminator, like the GAM part where they are taking this real image and the generated image here to make sure that they look plausible, realistic. Okay, so that was the pipeline, a bunch of components, seven neural networks here, as you can see, one, two, three, four, five, six, seven, five losses, and yeah, I actually forgot to mention these two. So let me just briefly go over those. So reconstruction loss, what I do is they make sure that this generated image here is close to this image here. So yeah, and that's the part which I suspect is causing the blurriness we saw in the results. The second part is the cyclic consistency loss from the CycleGAM paper, and this is a bit more complicated to explain, but what they do is the following. So they take this image, okay, they generate the output image, so this is Earth. So what they do now is they take this generated image, they blend it back into this original image because you know they have the bounding box, so they blend it in here, and then they pass that image again through the whole pipeline and generate another image, let's call that one OSC one prime, whatever. And now they want to make sure that that image is the same as this one, so they just do L1 between those two. So again, it's complicated, a bunch of details, but yeah, that's what I do, that's the whole pipeline, and now let's start digging into details a bit more. Okay, so let's start with encoders. So given a character string C for content, we therefore render it using a standard font for the text Verily Serif Mono on a plain white background producing the image 64 height and width is arbitrary, whereas cap denotes this standard font. Okay, so that's the part I explained to you, so this is this part. So you have the ground truth text, you just render the images, and the reason they do that is just so that they have uniform encoders here so that they don't have to encode text in some fancy ways, they just render it. So the style encoder is given a localized scene image along with a word bounding box denoting the word image from which the style is learned. So that's the part I explained to you here. So basically, this is the word image and they're actually including additional context so that it can better model the background, etc. Okay, and the penultimate layer of style encoder uses a region of interest, a line operator as described in mask rcnn to pull features coming from the desired word region, thereby reducing the style tensor to a fixed 512 dimensional representation. So that's the part I explained to you here because they have the bounding box, this yellow thing, they can use that information to kind of pull over this feature maps to get this final style vector. So that's the part about encoders. So again, they are resonant three, four, nothing fancy there. Now let's go to generator and style mapping network and let me explain you those. So the content representation is directly fed as input to the first layer of the generator instead of providing the learn tensor which was used in the original style again to model. So they just kind of explain some modifications they did, but it's basically still style style again, be too with some small modifications. And we introduce a style mapping network M which converts the style vector. Yes. So that's yes, is this thing here? Yes. Two layers specific style representations WSI where I indexes the layers of the generator, which are then fed as the adaptive instance normalization coefficient to each layer of the generator. So those, those WSI are just these things here. Okay. So now a quick detour into what adaptive instance normalization is. It's fairly simple. The original idea came from the neural style transfer literature. And what I do is the following. So imagine you have, imagine you have something like a VGG network and it's pre-trained on ImageNet, like it's a classifier and the weights are frozen. And what they do now is the following in that original paper. So they just take the style image, they pass it through this thing. They basically cache the feature maps or more precisely they cache the means and standard deviations of each of those feature maps. So once you have that, they input the content image here. And so what I do is if we take a specific content image feature map from a specific layer like this one, they're going to, as you can see here, find the mean statistic of the feature map, find the standard deviation and that's the just the instance normalization part. So they just use, so they have a feature map here and just going over the height and width features, they're going to find these mus and sigmas. They're going to normalize the content feature map and they're going to add the corresponding standard deviation and mean from the style image. So from this one. So those are the cached values. And by doing that, by just modifying those simple first order and first order statistics, they can basically, they show that they can adaptively change the style of the image. And yeah, so that was the basic idea and that's what style again was using and that's what this paper is also using to kind of generate the source style. Okay. Finally, I mentioned masks and they say here, the final mask provides a soft semantic segmentation of the generated image and is later used to construct the loss functions. Yeah, we'll see how they are exactly used. I already explained that this basically this recognizer is using masks as well as these generated images to do cross entropy over them and to train the generator, but they're also used for these reconstruction and cyclic loss and we'll see that in a minute. Okay. Let me, let me explain to you how this text perceptual loss works in a bit more detail. So specifically we assume a pre-trained type phase classification of proxy. Importantly, we make no assumption that the fonts see was trained on, trained to classify would appear in the photos we use to train and evaluate our textile brush. Okay. Now let me try and debug these equations for you. So it looks like this. Again, we have, let's say we have a VGG and in each layer we're gonna, so these are just the activation volumes and the spatial extent kind of goes down, whereas the number of channels goes up. So let me kind of depict it like this. So, so this is maybe the input image and then we're processing it. We get the feature maps and like, so this is the, this would be one feature map. This will be the second feature map. This would be the third feature map, et cetera. So this is the number of channels and we don't see the spatial extent of the image. I'm looking from it sideways. So what I do is they first do this. This is the perceptual loss part. So because this thing is pre-trained, it learns to extract certain features and they do the following. So they make sure that the generated image, so this is one, the output image and the input image are really close in the feature space. So they take certain feature maps, like maybe, I don't know, maybe take a certain subset here and we make sure that the, that those are really similar looking at L1 norm. Okay. So you take the generated image and you take the ground truth image and you want to make sure that these here, the features are really close, where closeness is defined by L1. That's the, that's the content loss in neural style transfer literature. The second thing they do is they form grand matrices also from those features. So you take those feature maps there and you do dot product between them pairwise and you end up with a certain thing called a feature map, which will have, so if this is C and they just do dot product between every single feature map, they'll end up with C times C matrix, which kind of captures the style of your image. And again, they want to make sure that the generated image, so this one, the generated one, as well as the input image have the same gray matrix. So they want to make them as close as possible again using L1. And this embedding thing is pretty much the same as this thing here. The only difference is that this psi here comes from the pen ultimate layer. So it's much deeper layer. So it's focusing on certain other features compared to these phi's here. And that's it in a nutshell. Again, I'm going to link a neural style transfer video somewhere here. You can check those out if you're not familiar with this, but this is pretty simple things. Once you wrap your head around it, it's pretty, pretty easy. Okay. They say here, note that the perceptual losses are only computed for the output image corresponding to original content C1. So they can't do it for the C2 word, so they can't do it for this one, for this warming one, because they don't have the actual ground truth image. So they just do it for the earth, for that one. Okay. That was that part of the loss. Now let's focus on the second part. We use the output string estimated by the recognizer module to compute a loss LR, which reflects how well the generator captured the desired content string C1, C2, simultaneously on both, as I said, so on both target images, on both the generated images and their masks. So how this will look like is the following. So again, let's take an input image, and this is something we generated. And we pass it through this recognizer network. And it will start generating character by character. And assuming the ground truth word is earth, as an example above, so earth. And assuming this one, the first distribution it generates looks something like this, like maybe like this. And it turns out that this argmax corresponds to letter B, whereas this one here maybe corresponds to letter E. And so obviously we made a mistake here. And what they do is just, they have the ground truth, the one hot encoding, they'll just do basically whatever this probability is, let's call it Pi. They're just going to do minus log of Pi. And as you can see, as Pi goes to one, which should be, which is the ground truth, the loss goes to zero. So this loss looks like this classic stuff. So this is one here, this is zero here, this is the loss, and this is the output probability. And they just repeat that process. So now they are going to feed this back into, and they're going to start generating next character, and next character, and next character. And basically they'll just be doing cross entropy and summing those and use that to back prop and train the generator. So that's how the text content loss works like. And now let's focus on the reconstruction losses. So here LREC represents the differences between the output generated image in style S and content C1, okay? And the cropped input style example ISC1. So basically, I think explained this, but let me go back here. So they're going to do L1 between this thing here, so this earth and this one. And because they're doing L1, I'm pretty sure this is causing them this blurriness problem they have. Yep. Okay. Finally, we compute a fake style vector ES' and generate this image. So let me try and draw this. Basically, so this is the input image, and this is the bounding box. So everything around here is the context, okay? So this is our initial input image. And in this example here, that will be earth, exactly this one. And now what I do is they feed this through the whole encoder generator pipeline and they generate the initial image. Let's call it O. And so what they now do is they take this O and they basically stitch it back in instead of EI here. So they'll have something like this. They'll have the same image, except that instead of I, they're going to blend in O. And again, they're going to pass this thing through the whole encoder generator image and they're going to generate another image called O'. And finally, they're going to do L1 between these two. So this one and this one. Just simple L1. And that's the cycle consistency loss from the cycle GAN paper, although there they use two different generator networks to do the same thing. So the semantics behind this approach is the following. So whatever we generate here, so this O, we want to make sure that after passing it another time through the whole encoder generator pipeline, we want to make sure that we can go back to the original word image I. That's the reasoning behind it. And yeah, you should go ahead and check out the cycle GAN paper to understand this a bit better. Again, we can use an L1 loss criterion for both reconstruction losses and split these functions on the foreground and background region separately using the generated masks. So this is where the masks come into play. This allows the network to learn fine-grained variations of the style present in the foreground region. Okay. And because they don't have any ground truth for masks, by doing this loss and by doing the previous loss with the recognizer module, they actually learn how to produce nice and accurate soft masks. That's pretty much it. Now, what they do is they just do a weighted average of all of these. And I can just imagine how the hyperparameter search for this looks like. Yeah. Additionally, the deep part is the discriminator-based adversarial loss. They additionally use some regularizations like the R1 regularization and path length regularization. I won't cover those into detail here, but just some regularization methods. R1 just kind of constrains the discriminator gradients, et cetera. So, yep, that was pretty much it for the model itself. So it's a fair bit of work here to make this work, I guess. And there are many, many different papers which are, which you need to read to understand everything here. So as I said, style GAN, neural style transfer, literature, then GANs in general, like OCR. So a bunch of components stitched together to give this pipeline the functionality it has. Okay. Now, other than the model, they also introduced this Imgur 5K handwriting set. And the whole point is it's got more variance, it's got much more diversity in the handwriting. The style variance is much bigger than looking at all of the other previous datasets. And they kind of show certain metrics here, like words, number of writers. You can see that there are much more writers here. The styles are whatever high means. Basically, there are more styles and there is background, there is more, it's more in the wild compared to the other datasets. Okay. Now, finally, let's focus on the results. We saw the visual results and they also did some ablations here where they tried to kind of remove certain components and just make sure that everything they are using is actually improving the performance. They're using metrics like MSC, so mean, square error, this structural similarity index measure, and this is just the signal to noise ratio metric. And finally, Fraschay's inception distance. So the three of these can only be used if they have the actual ground truth target image, which they have only for certain synthetic tests. So basically, FID is the one we should care about because that one kind of tries to capture what we humans think of a nice image. It's not perfect. Same thing goes for the inception score, but I think if used properly, it correlates nicely with human judgment. And they show here again, just using the discriminator or the GAN loss, they get poor performance, adding the recognizer module, so the OCR part improves everything, getting the reconstruction loss, and finally the cycle loss additionally gives them boost in all the metrics. So the interesting part here is that actually, once they add this final perceptual loss here, they boost the FID score, but they kind of lose on these objective metrics like MSC, SSAM, and PSNR, which are shown, by the way, not to correspond really good with our judgment. The reason being is, imagine you have, for MSC, if you have an image, and so if you generate an image and you just shift it by, I don't know, maybe five pixels downwards, so you just subtract five, you'll have an image which any human would say is perfect, but this MSC loss would be huge. So it just can't differentiate between salient regions and it just uses this dumb difference, like heuristic, to kind of give its judgment, but that doesn't correspond with human judgment that well. So what they do here is they try and see how good, this is a way to describe how good their generated images are by just passing them through this OCR engine. And that OCR engine is far from perfect, as you can see here, basically its accuracy drops significantly on this text visual question-answer dataset, but it's good enough and they try and feed the generated images from these two baselines as well from the text style brush paper, from this paper, and they notice that they have much higher accuracy. Now I'm a bit suspicious about these results because they were basically using that very same module to train the generator, so they kind of basically, the model can learn how to trick the recognizer, in a way can learn how to trick the recognizer module, I think here, so I'm not sure whether this is a fair comparison, but still, nonetheless, they did compare them. Okay, and here are some additional images where they've done the Poisson blending I mentioned, so again, I think the results are pretty impressive. If you take a look at this one later, Waste, this pretty much looks like an original image, it's pretty nice results there, as well as here, like awesome results, I don't see any problems with these ones, they later show some failure cases, I'll show you those in a bit. Again, they compare with a certain baseline, this time looking at the FID, and this is some other metric, I think it's called geometric something, I forgot, but basically, similar thing to FID, and they show that they get much lower results compared to the baseline, and again, lower is better for the FID metric. Okay, aside from these quantitative results, they did some user study where they just showed that this paper is better compared to the baselines, and as you can see here, so Davis and collaborators, the previous art, the previous work basically constrains itself to specific domain like handwriting or scene text, whereas this method just works on all of these, and again, it outperforms, looking at these user study, it outperforms the baseline, so both the SRNet and this one, and yeah, as well as side by side. Okay, so they show different failure cases here, and they're of very different natures, so here you can see that it fails to capture this italic style of handwriting, and as well as the thickness of the strokes, and yeah, and there are different failure cases, like here, it fails to capture these foreground effects, and I guess one of the main reasons is they are using those pre-trained recognizer, and modules, as well as the font classifier, combined that with the L1 reconstruction loss, I think all of those kind of jointly lead to this thing failing on these more complex styles like this calligraphy here, you can notice it here, or in this example, they lose this perspective distortion, let's call it, or maybe, I don't know, like this just, yeah, whatever, and yeah, I think that's pretty much it. They finally mentioned that this can be used in many cool applications like MR, like mixed reality and stuff, so our method aims at use cases involving creative self-expression and augmented reality, and that's something I'm really passionate about since I've been working on Microsoft HoloLens in the past, and they mentioned here, so photorealistic translation, so once you have some augmented reality glasses, you can imagine how you're looking at some language, like some foreign language, and all of a sudden, it's automatically translated and perfectly blended in as if it was real, and you can just kind of bridge this communication gap using that. They say here, our method can be used for data generation and augmentation for training future OCR systems, so that one is pretty obvious, so you can, obviously, when you can do this really good, you can use the data as a, basically, a new data point in a data set to train some OCR models by doing this, or just to augment an existing data set. However, they say here, we are aware that, like other technologies, ours can be misused, possibly the same as deepfake faces can be used for misinformation. So, deepfakes, that's a real concern for the society. I guess by just kind of open sourcing the data set and the research they're doing, they're kind of helping it, but at the end, it's just a game of cat and mouse. So, yeah, researchers are developing better tools, but by open sourcing this, you're also providing better tools for those who want to exploit it, so I don't think there is a great strategy to kind of cope with these deepfakes in general. Yeah, as a conclusion, they get some really cool results. They do have some failure cases. They do have problems, they still have problems with blur, but like all in all, it looks like great work, and yeah, maybe one of the things that focus on would be to try and reduce the number of components in the system. I guess it's pretty cumbersome to, if you want to change anything, you'd have to, you'd probably have, because all of these are coupled, you'd have to change a bunch of different values in order to retrain this thing. So I guess that would be a future research direction, and having said that, hopefully you liked this video. If you did, subscribe, share it out and until next time, bye bye.
[{"start": 0.0, "end": 5.0, "text": " What's up? In this video I'm covering this new paper called Textile Brush Transfer of"}, {"start": 5.0, "end": 11.120000000000001, "text": " Text Aesthetics from a Single Example by Pravin Krishnan, Ramakavuri, Guampeng, Boris"}, {"start": 11.120000000000001, "end": 17.94, "text": " Vasilev and Tal Hessner from Facebook AI Research. Basically they got some really nice results"}, {"start": 17.94, "end": 23.96, "text": " of rendering text in a specified style, in a one-shot setting. And let me show you what"}, {"start": 23.96, "end": 29.8, "text": " I mean by that. So basically you can see here in the source image you have some text and"}, {"start": 29.8, "end": 35.480000000000004, "text": " they kind of find a bounty box around it and they extract it and you can see just using"}, {"start": 35.480000000000004, "end": 44.22, "text": " this single word image they can learn the style and render novel content in that very"}, {"start": 44.22, "end": 49.82, "text": " same style one-shot. So basically just a single image like this one and you can do something"}, {"start": 49.82, "end": 56.480000000000004, "text": " like this. So they just kind of swapped a soya bottle for a tea vessel. And not only that,"}, {"start": 56.48, "end": 61.36, "text": " so you can see here another example on the book and not only that but they can also work"}, {"start": 61.36, "end": 66.16, "text": " cross-domain. So they're not constrained to a scene text editing domain. They can also"}, {"start": 66.16, "end": 72.03999999999999, "text": " do handwriting. So again here you can see a single word image and they can render novel"}, {"start": 72.03999999999999, "end": 78.75999999999999, "text": " sentences using that very same style. So the thing is they only generate these word images"}, {"start": 78.75999999999999, "end": 83.03999999999999, "text": " and how they actually blend these together is using something called Poisson blending."}, {"start": 83.04, "end": 86.64, "text": " That's the only part that feels a bit like hand crafting compared to this deep learning"}, {"start": 86.64, "end": 91.48, "text": " approach of actually generating these images. So let me show you some more examples what"}, {"start": 91.48, "end": 98.88000000000001, "text": " they achieve and then we'll start explaining the paper. So here you can see some examples"}, {"start": 98.88000000000001, "end": 105.88000000000001, "text": " on the left you have word image and on the right you can see the novel content in the"}, {"start": 105.88000000000001, "end": 111.94000000000001, "text": " same style. And it's pretty nice. There are some problems obviously here for example background"}, {"start": 111.94, "end": 118.92, "text": " is not the same. Like words are kind of blurry and I'll later give my hypothesis why that"}, {"start": 118.92, "end": 123.16, "text": " is and I think it's because of the reconstruction loss they're using with L1. But it's still"}, {"start": 123.16, "end": 128.16, "text": " pretty decent. So again here you can see the foreground texture is kind of lost here. The"}, {"start": 128.16, "end": 133.68, "text": " letters are kind of blurred. But all in all like it's really nice, looks awesome. Also"}, {"start": 133.68, "end": 138.84, "text": " sometimes I notice that the perspective is not the same so they don't manage to capture"}, {"start": 138.84, "end": 146.56, "text": " the sheer, the perspective of the source style. But like it's pretty awesome. Like look at"}, {"start": 146.56, "end": 152.56, "text": " this one. Disarming statement. I wouldn't be able to kind of discriminate between the"}, {"start": 152.56, "end": 158.18, "text": " two. They look the same to me. So nice results. Now let me start digging into how it works."}, {"start": 158.18, "end": 163.52, "text": " So they say here we present a novel approach for disentangling the content of a text image"}, {"start": 163.52, "end": 168.8, "text": " from all aspects of its appearance. The appearance representation we derive can then be applied"}, {"start": 168.8, "end": 174.44, "text": " to new content for one shot transfer of the source style to new content. We'll learn this"}, {"start": 174.44, "end": 179.08, "text": " disentanglement in a self-supervised manner. So this is not completely true because they"}, {"start": 179.08, "end": 183.88000000000002, "text": " actually have a couple of pre-training networks like one for, like one is an OCR engine, one"}, {"start": 183.88000000000002, "end": 190.20000000000002, "text": " is a font classifier and they also assume to have those bounding boxes as well as the"}, {"start": 190.20000000000002, "end": 195.32000000000002, "text": " ground truth text inside of those bounding boxes. So there, it's a combination of supervised"}, {"start": 195.32, "end": 200.28, "text": " learning and self-supervised learning, but like a nice portion of the pipeline is actually"}, {"start": 200.28, "end": 205.23999999999998, "text": " self-supervised. So that's cool. So our method processes entire word boxes without requiring"}, {"start": 205.23999999999998, "end": 210.12, "text": " a segmentation of text from background per character processing or making assumptions"}, {"start": 210.12, "end": 217.56, "text": " on string lengths. So that's cool. So as a kind of motivation for one shot learning,"}, {"start": 217.56, "end": 222.5, "text": " they say it here, moreover, it's unreasonable to expect text images used as style samples"}, {"start": 222.5, "end": 227.32, "text": " to provide example appearances for the entire alphabet, including digits. Transferring styles"}, {"start": 227.32, "end": 232.56, "text": " between strings with different characters therefore implies hallucinating plausible appearances"}, {"start": 232.56, "end": 238.96, "text": " of unseen characters in the input style. And that kind of, yeah, should associate you with"}, {"start": 238.96, "end": 243.28, "text": " generative adversarial networks and that's one of the components they're obviously using"}, {"start": 243.28, "end": 248.44, "text": " here in this work, but there are many components to it. And let me now kind of try and explain."}, {"start": 248.44, "end": 253.48, "text": " So I'll first try to give you like a high level overview and then we'll start digging"}, {"start": 253.48, "end": 257.71999999999997, "text": " into every single component. And like most of this video, we'll focus on explaining how"}, {"start": 257.71999999999997, "end": 263.16, "text": " this thing works because the problem itself is ill-posed as every, like similarly to neural"}, {"start": 263.16, "end": 269.68, "text": " style transfer because as a final conclusion, like humans just make a decision whether something"}, {"start": 269.68, "end": 274.04, "text": " looks nice or not. Or although there are some objective metrics like Fraschay's inception"}, {"start": 274.04, "end": 280.88, "text": " distance or inception score used in GANs, but like those are not that robust. And yeah,"}, {"start": 280.88, "end": 287.84000000000003, "text": " what they did also was a user study. So that's additionally, they've showed that this work"}, {"start": 287.84000000000003, "end": 294.6, "text": " looks better for a certain group of people they took for this experiment. So let me now"}, {"start": 294.6, "end": 301.40000000000003, "text": " start kind of dissecting this. So first things first, what they do is they take the image"}, {"start": 301.4, "end": 307.56, "text": " that has the style, like the style they want, and they crop it around the image word. So"}, {"start": 307.56, "end": 313.59999999999997, "text": " they put some context around the actual text that has the style they want. And that's one"}, {"start": 313.59999999999997, "end": 318.28, "text": " input. The second thing they do is they have a text which they want to render in the style"}, {"start": 318.28, "end": 324.03999999999996, "text": " from this image. And so you can see here, so earth is the, so they just take the word"}, {"start": 324.03999999999996, "end": 328.06, "text": " earth, which is the same as here, because as I told you, they have the ground truth,"}, {"start": 328.06, "end": 333.96, "text": " but they add some novel text for which they don't have the actual image. And what they"}, {"start": 333.96, "end": 340.32, "text": " then do is they don't input the text itself. They actually render the text on a white background"}, {"start": 340.32, "end": 344.92, "text": " and using some standard font. And we'll see those details a bit later. So once they render"}, {"start": 344.92, "end": 351.12, "text": " the text, now they can encode the text using this content encoder and they can encode the"}, {"start": 351.12, "end": 357.96, "text": " style using this style encoder. Both of these are just simple ResNet, basically ResNet,"}, {"start": 357.96, "end": 363.96, "text": " 34. And it's just modified a little bit. And so what I do here is because they have, as"}, {"start": 363.96, "end": 370.84, "text": " I told you, this bounding box ground truth, what I do is they take specific features corresponding"}, {"start": 370.84, "end": 377.21999999999997, "text": " to the bounding box and they do an average pooling along those pixels, along those features,"}, {"start": 377.21999999999997, "end": 381.35999999999996, "text": " and they create this vector here, which is a style vector, which was like, I think, 512,"}, {"start": 381.35999999999996, "end": 387.12, "text": " it doesn't matter, but yeah, let's make it a bit concrete. The thing they do with content"}, {"start": 387.12, "end": 392.16, "text": " is they just create a tensor here. Like as I said, they modify the ResNet 34, so they"}, {"start": 392.16, "end": 397.6, "text": " take just a certain feature maps from a certain layer. And the next thing they do is, you"}, {"start": 397.6, "end": 403.32, "text": " can see here, they feed those into this generator, which is style again, modified generator pretty"}, {"start": 403.32, "end": 410.92, "text": " much. So this style mapping network, what it does is the following. So it maps this"}, {"start": 410.92, "end": 417.76, "text": " style representation, this latent vector here, into multiple coefficients. And what these"}, {"start": 417.76, "end": 422.72, "text": " do if you're familiar with either the original adaptive instance normalization paper or with"}, {"start": 422.72, "end": 428.84000000000003, "text": " style again, is they kind of modify the feature maps, like they modify certain statistics"}, {"start": 428.84000000000003, "end": 436.02000000000004, "text": " like the mean and standard deviation so that they can kind of build in the style of their"}, {"start": 436.02, "end": 441.88, "text": " choosing, like the style they want. Okay? So I'll later explain how that adaptive instance"}, {"start": 441.88, "end": 445.71999999999997, "text": " normalization works, but for now you just have some coefficients here, which we got"}, {"start": 445.71999999999997, "end": 451.47999999999996, "text": " by mapping it from this style vector, and they modified, they use those to modify the"}, {"start": 451.47999999999996, "end": 457.0, "text": " feature maps. Kind of going, zooming in a little bit here, you can see the architecture"}, {"start": 457.0, "end": 464.24, "text": " is quite detailed. There are many components here, but the main idea is to use this modulated"}, {"start": 464.24, "end": 470.24, "text": " convolutions, so to use these coefficients to kind of modify the feature maps, and then"}, {"start": 470.24, "end": 476.92, "text": " they just have a progressive growing of these feature maps until they generate as the output"}, {"start": 476.92, "end": 482.04, "text": " this word image as well as the soft mask. So they're generating those two, so both the"}, {"start": 482.04, "end": 490.72, "text": " soft masks as well as the actual image words. So, okay. So once they generate those, they"}, {"start": 490.72, "end": 496.92, "text": " have five losses in total. So as you can see, this is basically hand crafting 2.0. They're"}, {"start": 496.92, "end": 502.32000000000005, "text": " using just DL components to create these systems, but the thing I don't like is that there are"}, {"start": 502.32000000000005, "end": 508.68, "text": " so many moving parts here, and I hope that like a couple of papers down the line, this"}, {"start": 508.68, "end": 513.9200000000001, "text": " will be something like end to end and we'll do the same thing. Okay, rent over. Let me"}, {"start": 513.9200000000001, "end": 520.1600000000001, "text": " explain to you briefly these losses functions. So they have, so these two here, so the typeface"}, {"start": 520.16, "end": 524.88, "text": " classifier and the recognizer are pre-trained and frozen during the training. So what I"}, {"start": 524.88, "end": 533.6, "text": " do is they take a, so this is a VGG 19 network, so this one is VGG 19. What I do is they take"}, {"start": 533.6, "end": 541.24, "text": " a VGG 19 network and they have some synthetic data set with certain image words, and they"}, {"start": 541.24, "end": 545.76, "text": " have actually the ground truth labels for the font, for the font type. So they just"}, {"start": 545.76, "end": 551.8, "text": " train a simple classifier that can predict what's the font of the input image. And a"}, {"start": 551.8, "end": 557.52, "text": " thing worth mentioning here is that the fonts used in that data set won't be the same as"}, {"start": 557.52, "end": 563.36, "text": " the fonts that this model will eventually be using because like they're using real data"}, {"start": 563.36, "end": 569.16, "text": " and here they just have synthetic fonts. So yeah, you can't cover the whole variance like"}, {"start": 569.16, "end": 575.4, "text": " that's present in real data sets. So having said that, what I do here is something, so"}, {"start": 575.4, "end": 579.24, "text": " this is a perceptual loss. If you're familiar with neural cell transfer, what they do here"}, {"start": 579.24, "end": 584.72, "text": " is they basically make sure, so as you can see, they input the ground truth, so this"}, {"start": 584.72, "end": 590.64, "text": " is this image, red image, and they input this generated image. And what I want to make sure"}, {"start": 590.64, "end": 596.16, "text": " is that the feature maps, it's a specific layer, are the same looking at one distance,"}, {"start": 596.16, "end": 601.36, "text": " for example. And also they form gram matrices which kind of capture the style of the image"}, {"start": 601.36, "end": 606.28, "text": " and they also make sure that those look the same for both this generated image as well"}, {"start": 606.28, "end": 611.4, "text": " as for this ground truth image. They also have some embedding loss, we'll get to that"}, {"start": 611.4, "end": 615.96, "text": " a bit later, but that's pretty much a neural style transfer part. Then they have recognizer"}, {"start": 615.96, "end": 621.5600000000001, "text": " which is a simple OCR engine. Again, they trained it, they took some pre-trained OCR"}, {"start": 621.5600000000001, "end": 627.2, "text": " engine, off the shelf thing, and what they do is because we have the ground truth text,"}, {"start": 627.2, "end": 632.2, "text": " we know that this is earth and warming, they just input all of these four, so both the"}, {"start": 632.2, "end": 638.1600000000001, "text": " masks as well as the generated images, and they kind of do the OCR. And finally they"}, {"start": 638.1600000000001, "end": 644.6400000000001, "text": " do simple cross entropy between the ground truth text and between whatever this OCR engine"}, {"start": 644.6400000000001, "end": 651.08, "text": " saw in these four images. So yeah, basically, let me just kind of explain that a bit more"}, {"start": 651.08, "end": 656.0400000000001, "text": " detailed. So let's say the first character, so it's out with certain distribution and"}, {"start": 656.04, "end": 660.8, "text": " you're gonna do argmax and that will be our first character. And so what they do is because"}, {"start": 660.8, "end": 665.0, "text": " they have the ground truth words, for example, this is the, let's say we're looking at this"}, {"start": 665.0, "end": 669.52, "text": " image, so they know that the first letter is E, so that will be one hot encoding of"}, {"start": 669.52, "end": 674.0, "text": " E, and they'll just do cross entropy between the output distribution and between the target"}, {"start": 674.0, "end": 679.88, "text": " distribution to train the generator. So again, this is frozen, so they are not modifying"}, {"start": 679.88, "end": 684.12, "text": " these weights, they are modifying the generator and the rest of the pipeline, okay? Same thing"}, {"start": 684.12, "end": 690.36, "text": " for this network here. And finally they have the, like this is pretty much default thing,"}, {"start": 690.36, "end": 697.12, "text": " discriminator, like the GAM part where they are taking this real image and the generated"}, {"start": 697.12, "end": 705.64, "text": " image here to make sure that they look plausible, realistic. Okay, so that was the pipeline,"}, {"start": 705.64, "end": 711.88, "text": " a bunch of components, seven neural networks here, as you can see, one, two, three, four,"}, {"start": 711.88, "end": 717.6, "text": " five, six, seven, five losses, and yeah, I actually forgot to mention these two. So let"}, {"start": 717.6, "end": 722.8, "text": " me just briefly go over those. So reconstruction loss, what I do is they make sure that this"}, {"start": 722.8, "end": 729.6, "text": " generated image here is close to this image here. So yeah, and that's the part which I"}, {"start": 729.6, "end": 736.4, "text": " suspect is causing the blurriness we saw in the results. The second part is the cyclic"}, {"start": 736.4, "end": 741.72, "text": " consistency loss from the CycleGAM paper, and this is a bit more complicated to explain,"}, {"start": 741.72, "end": 747.0, "text": " but what they do is the following. So they take this image, okay, they generate the output"}, {"start": 747.0, "end": 752.0, "text": " image, so this is Earth. So what they do now is they take this generated image, they blend"}, {"start": 752.0, "end": 757.0, "text": " it back into this original image because you know they have the bounding box, so they blend"}, {"start": 757.0, "end": 761.28, "text": " it in here, and then they pass that image again through the whole pipeline and generate"}, {"start": 761.28, "end": 769.12, "text": " another image, let's call that one OSC one prime, whatever. And now they want to make"}, {"start": 769.12, "end": 775.08, "text": " sure that that image is the same as this one, so they just do L1 between those two. So again,"}, {"start": 775.08, "end": 779.32, "text": " it's complicated, a bunch of details, but yeah, that's what I do, that's the whole pipeline,"}, {"start": 779.32, "end": 786.2, "text": " and now let's start digging into details a bit more. Okay, so let's start with encoders."}, {"start": 786.2, "end": 790.5600000000001, "text": " So given a character string C for content, we therefore render it using a standard font"}, {"start": 790.5600000000001, "end": 798.6, "text": " for the text Verily Serif Mono on a plain white background producing the image 64 height"}, {"start": 798.6, "end": 804.6800000000001, "text": " and width is arbitrary, whereas cap denotes this standard font. Okay, so that's the part"}, {"start": 804.6800000000001, "end": 808.66, "text": " I explained to you, so this is this part. So you have the ground truth text, you just"}, {"start": 808.66, "end": 815.0, "text": " render the images, and the reason they do that is just so that they have uniform encoders"}, {"start": 815.0, "end": 821.24, "text": " here so that they don't have to encode text in some fancy ways, they just render it. So"}, {"start": 821.24, "end": 826.6800000000001, "text": " the style encoder is given a localized scene image along with a word bounding box denoting"}, {"start": 826.68, "end": 830.88, "text": " the word image from which the style is learned. So that's the part I explained to you here."}, {"start": 830.88, "end": 836.64, "text": " So basically, this is the word image and they're actually including additional context so that"}, {"start": 836.64, "end": 843.28, "text": " it can better model the background, etc. Okay, and the penultimate layer of style encoder"}, {"start": 843.28, "end": 848.2399999999999, "text": " uses a region of interest, a line operator as described in mask rcnn to pull features"}, {"start": 848.2399999999999, "end": 854.0999999999999, "text": " coming from the desired word region, thereby reducing the style tensor to a fixed 512 dimensional"}, {"start": 854.1, "end": 857.0400000000001, "text": " representation. So that's the part I explained to you here because they have the bounding"}, {"start": 857.0400000000001, "end": 862.6800000000001, "text": " box, this yellow thing, they can use that information to kind of pull over this feature"}, {"start": 862.6800000000001, "end": 867.28, "text": " maps to get this final style vector. So that's the part about encoders. So again, they are"}, {"start": 867.28, "end": 872.36, "text": " resonant three, four, nothing fancy there. Now let's go to generator and style mapping"}, {"start": 872.36, "end": 877.96, "text": " network and let me explain you those. So the content representation is directly fed as"}, {"start": 877.96, "end": 882.4, "text": " input to the first layer of the generator instead of providing the learn tensor which"}, {"start": 882.4, "end": 887.48, "text": " was used in the original style again to model. So they just kind of explain some modifications"}, {"start": 887.48, "end": 892.88, "text": " they did, but it's basically still style style again, be too with some small modifications."}, {"start": 892.88, "end": 899.48, "text": " And we introduce a style mapping network M which converts the style vector. Yes. So that's"}, {"start": 899.48, "end": 909.12, "text": " yes, is this thing here? Yes. Two layers specific style representations WSI where I indexes"}, {"start": 909.12, "end": 914.5600000000001, "text": " the layers of the generator, which are then fed as the adaptive instance normalization"}, {"start": 914.5600000000001, "end": 922.44, "text": " coefficient to each layer of the generator. So those, those WSI are just these things"}, {"start": 922.44, "end": 930.52, "text": " here. Okay. So now a quick detour into what adaptive instance normalization is. It's fairly"}, {"start": 930.52, "end": 936.64, "text": " simple. The original idea came from the neural style transfer literature. And what I do is"}, {"start": 936.64, "end": 941.36, "text": " the following. So imagine you have, imagine you have something like a VGG network and"}, {"start": 941.36, "end": 945.6, "text": " it's pre-trained on ImageNet, like it's a classifier and the weights are frozen. And"}, {"start": 945.6, "end": 949.3199999999999, "text": " what they do now is the following in that original paper. So they just take the style"}, {"start": 949.3199999999999, "end": 955.08, "text": " image, they pass it through this thing. They basically cache the feature maps or more precisely"}, {"start": 955.08, "end": 961.96, "text": " they cache the means and standard deviations of each of those feature maps. So once you"}, {"start": 961.96, "end": 968.84, "text": " have that, they input the content image here. And so what I do is if we take a specific"}, {"start": 968.84, "end": 973.36, "text": " content image feature map from a specific layer like this one, they're going to, as"}, {"start": 973.36, "end": 980.9200000000001, "text": " you can see here, find the mean statistic of the feature map, find the standard deviation"}, {"start": 980.9200000000001, "end": 986.08, "text": " and that's the just the instance normalization part. So they just use, so they have a feature"}, {"start": 986.08, "end": 992.4000000000001, "text": " map here and just going over the height and width features, they're going to find these"}, {"start": 992.4000000000001, "end": 997.88, "text": " mus and sigmas. They're going to normalize the content feature map and they're going"}, {"start": 997.88, "end": 1005.2800000000001, "text": " to add the corresponding standard deviation and mean from the style image. So from this"}, {"start": 1005.2800000000001, "end": 1010.44, "text": " one. So those are the cached values. And by doing that, by just modifying those simple"}, {"start": 1010.44, "end": 1016.72, "text": " first order and first order statistics, they can basically, they show that they can adaptively"}, {"start": 1016.72, "end": 1022.4000000000001, "text": " change the style of the image. And yeah, so that was the basic idea and that's what style"}, {"start": 1022.4000000000001, "end": 1029.24, "text": " again was using and that's what this paper is also using to kind of generate the source"}, {"start": 1029.24, "end": 1030.24, "text": " style."}, {"start": 1030.24, "end": 1035.64, "text": " Okay. Finally, I mentioned masks and they say here, the final mask provides a soft semantic"}, {"start": 1035.64, "end": 1040.72, "text": " segmentation of the generated image and is later used to construct the loss functions."}, {"start": 1040.72, "end": 1048.44, "text": " Yeah, we'll see how they are exactly used. I already explained that this basically this"}, {"start": 1048.44, "end": 1055.0400000000002, "text": " recognizer is using masks as well as these generated images to do cross entropy over"}, {"start": 1055.0400000000002, "end": 1059.16, "text": " them and to train the generator, but they're also used for these reconstruction and cyclic"}, {"start": 1059.16, "end": 1063.92, "text": " loss and we'll see that in a minute. Okay. Let me, let me explain to you how this text"}, {"start": 1063.92, "end": 1068.3600000000001, "text": " perceptual loss works in a bit more detail. So specifically we assume a pre-trained type"}, {"start": 1068.3600000000001, "end": 1073.0, "text": " phase classification of proxy. Importantly, we make no assumption that the fonts see was"}, {"start": 1073.0, "end": 1077.8400000000001, "text": " trained on, trained to classify would appear in the photos we use to train and evaluate"}, {"start": 1077.8400000000001, "end": 1087.52, "text": " our textile brush. Okay. Now let me try and debug these equations for you. So it looks"}, {"start": 1087.52, "end": 1096.08, "text": " like this. Again, we have, let's say we have a VGG and in each layer we're gonna, so these"}, {"start": 1096.08, "end": 1101.8799999999999, "text": " are just the activation volumes and the spatial extent kind of goes down, whereas the number"}, {"start": 1101.8799999999999, "end": 1109.08, "text": " of channels goes up. So let me kind of depict it like this. So, so this is maybe the input"}, {"start": 1109.08, "end": 1114.84, "text": " image and then we're processing it. We get the feature maps and like, so this is the,"}, {"start": 1114.84, "end": 1117.6799999999998, "text": " this would be one feature map. This will be the second feature map. This would be the"}, {"start": 1117.6799999999998, "end": 1122.9599999999998, "text": " third feature map, et cetera. So this is the number of channels and we don't see the spatial"}, {"start": 1122.9599999999998, "end": 1130.8, "text": " extent of the image. I'm looking from it sideways. So what I do is they first do this. This is"}, {"start": 1130.8, "end": 1137.04, "text": " the perceptual loss part. So because this thing is pre-trained, it learns to extract certain"}, {"start": 1137.04, "end": 1144.6399999999999, "text": " features and they do the following. So they make sure that the generated image, so this"}, {"start": 1144.64, "end": 1150.1200000000001, "text": " is one, the output image and the input image are really close in the feature space. So"}, {"start": 1150.1200000000001, "end": 1155.0, "text": " they take certain feature maps, like maybe, I don't know, maybe take a certain subset"}, {"start": 1155.0, "end": 1162.8000000000002, "text": " here and we make sure that the, that those are really similar looking at L1 norm. Okay."}, {"start": 1162.8000000000002, "end": 1168.48, "text": " So you take the generated image and you take the ground truth image and you want to make"}, {"start": 1168.48, "end": 1175.6, "text": " sure that these here, the features are really close, where closeness is defined by L1. That's"}, {"start": 1175.6, "end": 1182.16, "text": " the, that's the content loss in neural style transfer literature. The second thing they"}, {"start": 1182.16, "end": 1187.64, "text": " do is they form grand matrices also from those features. So you take those feature maps there"}, {"start": 1187.64, "end": 1193.76, "text": " and you do dot product between them pairwise and you end up with a certain thing called"}, {"start": 1193.76, "end": 1198.32, "text": " a feature map, which will have, so if this is C and they just do dot product between"}, {"start": 1198.32, "end": 1203.6, "text": " every single feature map, they'll end up with C times C matrix, which kind of captures the"}, {"start": 1203.6, "end": 1208.4, "text": " style of your image. And again, they want to make sure that the generated image, so"}, {"start": 1208.4, "end": 1214.72, "text": " this one, the generated one, as well as the input image have the same gray matrix. So"}, {"start": 1214.72, "end": 1219.68, "text": " they want to make them as close as possible again using L1. And this embedding thing is"}, {"start": 1219.68, "end": 1226.54, "text": " pretty much the same as this thing here. The only difference is that this psi here comes"}, {"start": 1226.54, "end": 1231.6000000000001, "text": " from the pen ultimate layer. So it's much deeper layer. So it's focusing on certain"}, {"start": 1231.6000000000001, "end": 1239.16, "text": " other features compared to these phi's here. And that's it in a nutshell. Again, I'm going"}, {"start": 1239.16, "end": 1242.64, "text": " to link a neural style transfer video somewhere here. You can check those out if you're not"}, {"start": 1242.64, "end": 1247.0800000000002, "text": " familiar with this, but this is pretty simple things. Once you wrap your head around it,"}, {"start": 1247.08, "end": 1253.28, "text": " it's pretty, pretty easy. Okay. They say here, note that the perceptual losses are only computed"}, {"start": 1253.28, "end": 1261.0, "text": " for the output image corresponding to original content C1. So they can't do it for the C2"}, {"start": 1261.0, "end": 1267.04, "text": " word, so they can't do it for this one, for this warming one, because they don't have"}, {"start": 1267.04, "end": 1273.04, "text": " the actual ground truth image. So they just do it for the earth, for that one. Okay. That"}, {"start": 1273.04, "end": 1278.72, "text": " was that part of the loss. Now let's focus on the second part. We use the output string"}, {"start": 1278.72, "end": 1285.12, "text": " estimated by the recognizer module to compute a loss LR, which reflects how well the generator"}, {"start": 1285.12, "end": 1291.58, "text": " captured the desired content string C1, C2, simultaneously on both, as I said, so on both"}, {"start": 1291.58, "end": 1296.12, "text": " target images, on both the generated images and their masks. So how this will look like"}, {"start": 1296.12, "end": 1302.6, "text": " is the following. So again, let's take an input image, and this is something we generated."}, {"start": 1302.6, "end": 1309.84, "text": " And we pass it through this recognizer network. And it will start generating character by"}, {"start": 1309.84, "end": 1316.8, "text": " character. And assuming the ground truth word is earth, as an example above, so earth. And"}, {"start": 1316.8, "end": 1320.84, "text": " assuming this one, the first distribution it generates looks something like this, like"}, {"start": 1320.84, "end": 1326.7199999999998, "text": " maybe like this. And it turns out that this argmax corresponds to letter B, whereas this"}, {"start": 1326.7199999999998, "end": 1332.04, "text": " one here maybe corresponds to letter E. And so obviously we made a mistake here. And what"}, {"start": 1332.04, "end": 1338.32, "text": " they do is just, they have the ground truth, the one hot encoding, they'll just do basically"}, {"start": 1338.32, "end": 1345.3999999999999, "text": " whatever this probability is, let's call it Pi. They're just going to do minus log of"}, {"start": 1345.3999999999999, "end": 1353.54, "text": " Pi. And as you can see, as Pi goes to one, which should be, which is the ground truth,"}, {"start": 1353.54, "end": 1359.18, "text": " the loss goes to zero. So this loss looks like this classic stuff. So this is one here,"}, {"start": 1359.18, "end": 1366.0, "text": " this is zero here, this is the loss, and this is the output probability. And they just repeat"}, {"start": 1366.0, "end": 1370.4, "text": " that process. So now they are going to feed this back into, and they're going to start"}, {"start": 1370.4, "end": 1377.76, "text": " generating next character, and next character, and next character. And basically they'll"}, {"start": 1377.76, "end": 1383.3600000000001, "text": " just be doing cross entropy and summing those and use that to back prop and train the generator."}, {"start": 1383.3600000000001, "end": 1388.4, "text": " So that's how the text content loss works like. And now let's focus on the reconstruction"}, {"start": 1388.4, "end": 1394.96, "text": " losses. So here LREC represents the differences between the output generated image in style"}, {"start": 1394.96, "end": 1403.24, "text": " S and content C1, okay? And the cropped input style example ISC1. So basically, I think"}, {"start": 1403.24, "end": 1408.72, "text": " explained this, but let me go back here. So they're going to do L1 between this thing"}, {"start": 1408.72, "end": 1413.6000000000001, "text": " here, so this earth and this one. And because they're doing L1, I'm pretty sure this is"}, {"start": 1413.6, "end": 1421.76, "text": " causing them this blurriness problem they have. Yep. Okay. Finally, we compute a fake"}, {"start": 1421.76, "end": 1430.6799999999998, "text": " style vector ES' and generate this image. So let me try and draw this. Basically, so"}, {"start": 1430.6799999999998, "end": 1438.0, "text": " this is the input image, and this is the bounding box. So everything around here is the context,"}, {"start": 1438.0, "end": 1445.0, "text": " okay? So this is our initial input image. And in this example here, that will be earth,"}, {"start": 1445.0, "end": 1452.68, "text": " exactly this one. And now what I do is they feed this through the whole encoder generator"}, {"start": 1452.68, "end": 1459.64, "text": " pipeline and they generate the initial image. Let's call it O. And so what they now do is"}, {"start": 1459.64, "end": 1465.28, "text": " they take this O and they basically stitch it back in instead of EI here. So they'll"}, {"start": 1465.28, "end": 1471.44, "text": " have something like this. They'll have the same image, except that instead of I, they're"}, {"start": 1471.44, "end": 1478.48, "text": " going to blend in O. And again, they're going to pass this thing through the whole encoder"}, {"start": 1478.48, "end": 1486.8, "text": " generator image and they're going to generate another image called O'. And finally, they're"}, {"start": 1486.8, "end": 1495.76, "text": " going to do L1 between these two. So this one and this one. Just simple L1. And that's"}, {"start": 1495.76, "end": 1503.9199999999998, "text": " the cycle consistency loss from the cycle GAN paper, although there they use two different"}, {"start": 1503.9199999999998, "end": 1508.84, "text": " generator networks to do the same thing. So the semantics behind this approach is the"}, {"start": 1508.84, "end": 1513.8799999999999, "text": " following. So whatever we generate here, so this O, we want to make sure that after passing"}, {"start": 1513.88, "end": 1518.96, "text": " it another time through the whole encoder generator pipeline, we want to make sure that"}, {"start": 1518.96, "end": 1528.0, "text": " we can go back to the original word image I. That's the reasoning behind it. And yeah,"}, {"start": 1528.0, "end": 1532.0800000000002, "text": " you should go ahead and check out the cycle GAN paper to understand this a bit better."}, {"start": 1532.0800000000002, "end": 1537.14, "text": " Again, we can use an L1 loss criterion for both reconstruction losses and split these"}, {"start": 1537.14, "end": 1541.92, "text": " functions on the foreground and background region separately using the generated masks."}, {"start": 1541.92, "end": 1546.04, "text": " So this is where the masks come into play. This allows the network to learn fine-grained"}, {"start": 1546.04, "end": 1552.1200000000001, "text": " variations of the style present in the foreground region. Okay. And because they don't have"}, {"start": 1552.1200000000001, "end": 1557.2, "text": " any ground truth for masks, by doing this loss and by doing the previous loss with the"}, {"start": 1557.2, "end": 1564.6000000000001, "text": " recognizer module, they actually learn how to produce nice and accurate soft masks. That's"}, {"start": 1564.6000000000001, "end": 1569.8000000000002, "text": " pretty much it. Now, what they do is they just do a weighted average of all of these."}, {"start": 1569.8, "end": 1577.48, "text": " And I can just imagine how the hyperparameter search for this looks like. Yeah. Additionally,"}, {"start": 1577.48, "end": 1585.52, "text": " the deep part is the discriminator-based adversarial loss. They additionally use some regularizations"}, {"start": 1585.52, "end": 1591.12, "text": " like the R1 regularization and path length regularization. I won't cover those into detail"}, {"start": 1591.12, "end": 1596.48, "text": " here, but just some regularization methods. R1 just kind of constrains the discriminator"}, {"start": 1596.48, "end": 1605.1200000000001, "text": " gradients, et cetera. So, yep, that was pretty much it for the model itself. So it's a fair"}, {"start": 1605.1200000000001, "end": 1614.8, "text": " bit of work here to make this work, I guess. And there are many, many different papers"}, {"start": 1614.8, "end": 1621.68, "text": " which are, which you need to read to understand everything here. So as I said, style GAN,"}, {"start": 1621.68, "end": 1629.92, "text": " neural style transfer, literature, then GANs in general, like OCR. So a bunch of components"}, {"start": 1629.92, "end": 1639.88, "text": " stitched together to give this pipeline the functionality it has. Okay. Now, other than"}, {"start": 1639.88, "end": 1644.92, "text": " the model, they also introduced this Imgur 5K handwriting set. And the whole point is"}, {"start": 1644.92, "end": 1653.4, "text": " it's got more variance, it's got much more diversity in the handwriting. The style variance"}, {"start": 1653.4, "end": 1657.96, "text": " is much bigger than looking at all of the other previous datasets. And they kind of"}, {"start": 1657.96, "end": 1662.92, "text": " show certain metrics here, like words, number of writers. You can see that there are much"}, {"start": 1662.92, "end": 1668.24, "text": " more writers here. The styles are whatever high means. Basically, there are more styles"}, {"start": 1668.24, "end": 1674.92, "text": " and there is background, there is more, it's more in the wild compared to the other datasets."}, {"start": 1674.92, "end": 1682.92, "text": " Okay. Now, finally, let's focus on the results. We saw the visual results and they also did"}, {"start": 1682.92, "end": 1689.04, "text": " some ablations here where they tried to kind of remove certain components and just make"}, {"start": 1689.04, "end": 1694.72, "text": " sure that everything they are using is actually improving the performance. They're using metrics"}, {"start": 1694.72, "end": 1702.72, "text": " like MSC, so mean, square error, this structural similarity index measure, and this is just"}, {"start": 1702.72, "end": 1708.3600000000001, "text": " the signal to noise ratio metric. And finally, Fraschay's inception distance. So the three"}, {"start": 1708.3600000000001, "end": 1716.56, "text": " of these can only be used if they have the actual ground truth target image, which they"}, {"start": 1716.56, "end": 1724.0, "text": " have only for certain synthetic tests. So basically, FID is the one we should care about"}, {"start": 1724.0, "end": 1729.24, "text": " because that one kind of tries to capture what we humans think of a nice image. It's"}, {"start": 1729.24, "end": 1737.72, "text": " not perfect. Same thing goes for the inception score, but I think if used properly, it correlates"}, {"start": 1737.72, "end": 1745.56, "text": " nicely with human judgment. And they show here again, just using the discriminator or"}, {"start": 1745.56, "end": 1751.68, "text": " the GAN loss, they get poor performance, adding the recognizer module, so the OCR part improves"}, {"start": 1751.68, "end": 1757.52, "text": " everything, getting the reconstruction loss, and finally the cycle loss additionally gives"}, {"start": 1757.52, "end": 1763.68, "text": " them boost in all the metrics. So the interesting part here is that actually, once they add"}, {"start": 1763.68, "end": 1772.5600000000002, "text": " this final perceptual loss here, they boost the FID score, but they kind of lose on these"}, {"start": 1772.5600000000002, "end": 1779.6000000000001, "text": " objective metrics like MSC, SSAM, and PSNR, which are shown, by the way, not to correspond"}, {"start": 1779.6, "end": 1784.9599999999998, "text": " really good with our judgment. The reason being is, imagine you have, for MSC, if you"}, {"start": 1784.9599999999998, "end": 1789.36, "text": " have an image, and so if you generate an image and you just shift it by, I don't know, maybe"}, {"start": 1789.36, "end": 1796.1599999999999, "text": " five pixels downwards, so you just subtract five, you'll have an image which any human"}, {"start": 1796.1599999999999, "end": 1802.9599999999998, "text": " would say is perfect, but this MSC loss would be huge. So it just can't differentiate between"}, {"start": 1802.96, "end": 1810.64, "text": " salient regions and it just uses this dumb difference, like heuristic, to kind of give"}, {"start": 1810.64, "end": 1815.16, "text": " its judgment, but that doesn't correspond with human judgment that well. So what they"}, {"start": 1815.16, "end": 1820.4, "text": " do here is they try and see how good, this is a way to describe how good their generated"}, {"start": 1820.4, "end": 1826.92, "text": " images are by just passing them through this OCR engine. And that OCR engine is far from"}, {"start": 1826.92, "end": 1833.0, "text": " perfect, as you can see here, basically its accuracy drops significantly on this text"}, {"start": 1833.0, "end": 1839.52, "text": " visual question-answer dataset, but it's good enough and they try and feed the generated"}, {"start": 1839.52, "end": 1846.88, "text": " images from these two baselines as well from the text style brush paper, from this paper,"}, {"start": 1846.88, "end": 1851.44, "text": " and they notice that they have much higher accuracy. Now I'm a bit suspicious about these"}, {"start": 1851.44, "end": 1858.24, "text": " results because they were basically using that very same module to train the generator,"}, {"start": 1858.24, "end": 1865.52, "text": " so they kind of basically, the model can learn how to trick the recognizer, in a way can"}, {"start": 1865.52, "end": 1869.04, "text": " learn how to trick the recognizer module, I think here, so I'm not sure whether this"}, {"start": 1869.04, "end": 1876.8, "text": " is a fair comparison, but still, nonetheless, they did compare them. Okay, and here are"}, {"start": 1876.8, "end": 1883.0, "text": " some additional images where they've done the Poisson blending I mentioned, so again,"}, {"start": 1883.0, "end": 1888.52, "text": " I think the results are pretty impressive. If you take a look at this one later, Waste,"}, {"start": 1888.52, "end": 1894.08, "text": " this pretty much looks like an original image, it's pretty nice results there, as well as"}, {"start": 1894.08, "end": 1899.76, "text": " here, like awesome results, I don't see any problems with these ones, they later show"}, {"start": 1899.76, "end": 1906.28, "text": " some failure cases, I'll show you those in a bit. Again, they compare with a certain"}, {"start": 1906.28, "end": 1911.3999999999999, "text": " baseline, this time looking at the FID, and this is some other metric, I think it's called"}, {"start": 1911.3999999999999, "end": 1917.12, "text": " geometric something, I forgot, but basically, similar thing to FID, and they show that they"}, {"start": 1917.12, "end": 1925.56, "text": " get much lower results compared to the baseline, and again, lower is better for the FID metric."}, {"start": 1925.56, "end": 1932.32, "text": " Okay, aside from these quantitative results, they did some user study where they just showed"}, {"start": 1932.32, "end": 1938.28, "text": " that this paper is better compared to the baselines, and as you can see here, so Davis"}, {"start": 1938.28, "end": 1945.8799999999999, "text": " and collaborators, the previous art, the previous work basically constrains itself to specific"}, {"start": 1945.8799999999999, "end": 1951.3999999999999, "text": " domain like handwriting or scene text, whereas this method just works on all of these, and"}, {"start": 1951.3999999999999, "end": 1957.04, "text": " again, it outperforms, looking at these user study, it outperforms the baseline, so both"}, {"start": 1957.04, "end": 1963.3999999999999, "text": " the SRNet and this one, and yeah, as well as side by side. Okay, so they show different"}, {"start": 1963.3999999999999, "end": 1970.6399999999999, "text": " failure cases here, and they're of very different natures, so here you can see that it fails"}, {"start": 1970.6399999999999, "end": 1976.68, "text": " to capture this italic style of handwriting, and as well as the thickness of the strokes,"}, {"start": 1976.68, "end": 1983.08, "text": " and yeah, and there are different failure cases, like here, it fails to capture these"}, {"start": 1983.08, "end": 1988.9199999999998, "text": " foreground effects, and I guess one of the main reasons is they are using those pre-trained"}, {"start": 1988.9199999999998, "end": 1996.9199999999998, "text": " recognizer, and modules, as well as the font classifier, combined that with the L1 reconstruction"}, {"start": 1996.9199999999998, "end": 2004.72, "text": " loss, I think all of those kind of jointly lead to this thing failing on these more complex"}, {"start": 2004.72, "end": 2011.8799999999999, "text": " styles like this calligraphy here, you can notice it here, or in this example, they lose"}, {"start": 2011.88, "end": 2017.6000000000001, "text": " this perspective distortion, let's call it, or maybe, I don't know, like this just, yeah,"}, {"start": 2017.6000000000001, "end": 2023.72, "text": " whatever, and yeah, I think that's pretty much it. They finally mentioned that this"}, {"start": 2023.72, "end": 2028.5600000000002, "text": " can be used in many cool applications like MR, like mixed reality and stuff, so our method"}, {"start": 2028.5600000000002, "end": 2034.48, "text": " aims at use cases involving creative self-expression and augmented reality, and that's something"}, {"start": 2034.48, "end": 2039.16, "text": " I'm really passionate about since I've been working on Microsoft HoloLens in the past,"}, {"start": 2039.16, "end": 2043.28, "text": " and they mentioned here, so photorealistic translation, so once you have some augmented"}, {"start": 2043.28, "end": 2047.3600000000001, "text": " reality glasses, you can imagine how you're looking at some language, like some foreign"}, {"start": 2047.3600000000001, "end": 2052.58, "text": " language, and all of a sudden, it's automatically translated and perfectly blended in as if"}, {"start": 2052.58, "end": 2059.44, "text": " it was real, and you can just kind of bridge this communication gap using that. They say"}, {"start": 2059.44, "end": 2064.6800000000003, "text": " here, our method can be used for data generation and augmentation for training future OCR systems,"}, {"start": 2064.68, "end": 2069.72, "text": " so that one is pretty obvious, so you can, obviously, when you can do this really good,"}, {"start": 2069.72, "end": 2076.12, "text": " you can use the data as a, basically, a new data point in a data set to train some OCR"}, {"start": 2076.12, "end": 2081.96, "text": " models by doing this, or just to augment an existing data set. However, they say here,"}, {"start": 2081.96, "end": 2086.68, "text": " we are aware that, like other technologies, ours can be misused, possibly the same as"}, {"start": 2086.68, "end": 2092.64, "text": " deepfake faces can be used for misinformation. So, deepfakes, that's a real concern for the"}, {"start": 2092.64, "end": 2097.2, "text": " society. I guess by just kind of open sourcing the data set and the research they're doing,"}, {"start": 2097.2, "end": 2102.92, "text": " they're kind of helping it, but at the end, it's just a game of cat and mouse. So, yeah,"}, {"start": 2102.92, "end": 2107.3599999999997, "text": " researchers are developing better tools, but by open sourcing this, you're also providing"}, {"start": 2107.3599999999997, "end": 2111.16, "text": " better tools for those who want to exploit it, so I don't think there is a great strategy"}, {"start": 2111.16, "end": 2118.08, "text": " to kind of cope with these deepfakes in general. Yeah, as a conclusion, they get some really"}, {"start": 2118.08, "end": 2122.7999999999997, "text": " cool results. They do have some failure cases. They do have problems, they still have problems"}, {"start": 2122.7999999999997, "end": 2129.52, "text": " with blur, but like all in all, it looks like great work, and yeah, maybe one of the things"}, {"start": 2129.52, "end": 2135.36, "text": " that focus on would be to try and reduce the number of components in the system. I guess"}, {"start": 2135.36, "end": 2140.44, "text": " it's pretty cumbersome to, if you want to change anything, you'd have to, you'd probably"}, {"start": 2140.44, "end": 2143.66, "text": " have, because all of these are coupled, you'd have to change a bunch of different values"}, {"start": 2143.66, "end": 2148.7599999999998, "text": " in order to retrain this thing. So I guess that would be a future research direction,"}, {"start": 2148.7599999999998, "end": 2154.04, "text": " and having said that, hopefully you liked this video. If you did, subscribe, share it"}, {"start": 2154.04, "end": 2174.44, "text": " out and until next time, bye bye."}]