Fake It Till You Make It (Microsoft) | Paper Explained
❤️ Become The AI Epiphany 
Patreon ❤️ ► https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

In this video I cover Microsoft's "Fake it till you make it: face analysis in the wild using synthetic data alone" paper.

They introduce synthetic (face) data that is realistic enough to enable trivial generalization to real data!

Disclaimer: I was a part of this team while back at Microsoft, and even had some contributions to this project, but in this video, I only talk about the stuff that can be deduced from the paper itself. 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2109.15102
✅ Demo video: https://www.youtube.com/watch?v=wlOMpQe8luQ&ab_channel=MicrosoftResearch
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 The domain gap problem
04:20 The paper deals only with human faces
06:05 Data-centric approach is short-sighted?
08:50 Building 3D human faces (high-level explanation)
09:30 Pros and cons of the approach
11:45 Building 3D human faces (in-depth)
16:15 Results on landmark localization and sem seg
20:45 Better generalization?
21:25 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#facesynthetics #syntheticdata #microsoft
 What's cracking guys in this video I'm covering this new paper called fake it till you make it face analysis in the wild using synthetic data alone by these awesome people from Microsoft and I actually had the privilege of working with them while I was back at Microsoft so as a quick disclaimer I actually did contribute a little bit to this framework the synthetics framework And so I know a lot of details of how this project was structured and everything in this video I'm gonna only tell you obviously only the things and conclusions you can you can deduce from the paper itself So having said that these guys are a real wood Tata spell through shade this Charlie Hewitt Sebastian Gaggio Matthew Johnson Virginia as tellers Thomas Cashman and Jamie Sutton, so let's see what the hype is all about They say here that we demonstrated it's possible to perform face related computer vision in the wild using synthetic data alone So I mean obviously you want to use synthetic see if that's possible because you have perfect ground truth But what was the problem so far well? there is something called domain gap and I think it's fairly obvious when you try and train your models on synthetic data and Then try to do to do inference with that same model on real data. You're gonna have some problems The reason is you have different distributions in the case where your synthetics is not perfect as it usually is not you basically have a gap because the distribution between the those data points from the synthetics is different from the distribution of the data points in the real data and So different people different groups try to solve this in different ways and some good examples would be like opening eye so I Recently a couple months ago covered this opening ice paper where they used the robotic hand to solve the Rubik's Cube and the thing Is the policy the RL agent was actually trained purely in simulation? The approach they took was to not to try to create a perfect simulation because it's even harder To create a perfect simulator for this robotic hand then to create a simulator For the for the faces because here you only need to mimic the appearance of these faces Whereas in the in this case you have to model the physics you have to model the friction you have to model the contact Forces which is super hard to do and so what they did is just Randomize the simulation so that the real world appears like just another instantiation of a simulation So they modify even stuff like gravity from the usual 9.81 at least on earth to like even 8.5 or whatnot, but it was all done gradually In order to have this curriculum like like a gradual training So that was one approach second approach other approaches were stuff like you can fine-tune On real images after training on your synthetics, so you have imperfect synthetics you fine-tune in real data from the target distribution And then you can expect that the the accuracy or whatever metric you care about is going to improve And finally there are approaches such as uprealing with GANs whereby you take your real data, and you pass them through certain like GANs which will make those images appear more realistic Those are all the different ways to try to reduce this domain gap so Microsoft on the other hand took another route and that's to reduce the domain gap almost down to zero And we have to have some bad jokes on this on this channel. So yeah anyways Basically, here are some of the images that the framework produces you can see they are fairly realistic Especially this this this this person here. You can see that you can have geometry So 3d information because you have a synthetic data and that means you also have perfect labeling, which is awesome Which is the superpower of this whole framework. You can do pixel wise like a semantic segmentation You have the landmarks however dense you want them to be and a lot of other Ground-truth data which we're going to see a bit later and before I dig deeper into the paper Let me just show you this quick clip from Microsoft where you'll see the animations of the of this of this syntax framework Here are some examples synthetic faces that we render as training data for machine learning With synthetic data you can guarantee perfect labels without annotation noise Generate rich labels that are otherwise impossible to label by hand and have full control over variation and diversity in a data set Awesome, so let's continue and dig deeper into this paper So they say here that they show that the synthetic data can match can both match real data inaccuracy as well as open up New approaches where manual labeling would be impossible. Okay, so a quick remark here. You need to keep in mind that rendering Only face images so the heads of of humans is a fairly constrained problem I mean you have Uh, like a certain geometry that all humans have we have usually have two eyes We usually have one nose and I mean the problem is very very constrained and I really believe that we can Actually reduce to the main gap almost down to zero if not to zero in the future For this particular problem that Microsoft deeply cares about so because this the team works in mixed reality and virtual reality so that means they they care about stuff like Microsoft HoloLens and There are other competitors out there such as Magic Leap and Facebook Reality Labs and all of them care about this very constrained problem of like understanding human faces and bodies and so That the point I'm trying to drive here is that if you had some other area such as like self-driving I'm fairly sure this is not the way I had because I mean it boils down to the question whether it's easier to build up a new world or just understand the The the the current world because in order to build a perfect simulator for self-driving you literally have to Cover so many edge cases, etc. So you literally have to build something like I guess real-world simulator and Yeah, I'm fairly sure that's harder than just trying to build a computer vision systems that will solve us self-driving continuing on here When they say that when faced with a machine learning problem the hardest challenge often isn't choosing the right machine learning model It's finding the right data and this reminds me strongly of Andrew Eng's data centric approach And the thing I noticed here is that all of these people and groups that are focused more on short-term products Tend to say something like this and I don't mean that in a in a in a bad way. I just think that like this implies obviously implies supervised learning paradigm and I'm fairly sure that's not the way that kids and humans learn and I'm fairly sure that approaches that for example Facebook and deep minor pushing Like self supervised paradigm is the way to go long term. But having said that again, I Think this is a fairly fairly good approach At the time being because we're gonna build better product which will enable us accelerate Accomplishing those long-term goals of cracking intelligence Obviously they mentioned some some benefits of synthetic data here So with synthetic data you can guarantee perfect labels without annotation noise Generate rich labels, which are otherwise Impossible to label by hand and they have an example later here in the paper. Let me show you basically, I mean you can you can generate As much landmarks as you want like you can you can make the the mesh as dense as you want and this is obviously not possible To achieve by by manual labeling it would be super expensive and super super error prone So that's some of the benefits that real data that synthetic data is bringing. Let me go back here Aside from that you have a full control over variation and diversity in a data set Which is also very very important when we are deploying these products globally You want to make sure it works on different demographics and they even have a histogram down here Let me quickly show you this so they show the histograms here over the age Over the I guess it's sex not gender and and ethnicity and as you can see here It's it's a work in progress obviously, but the good thing is you can basically improve this Whereas it's way harder to do this when you're trying to collect real data and I guess a small remark here. It'd be nice to have like a ground truth histograms here I the global statistics so that we can compare how far away is this current? Histogram from the actual desired target histogram. So this is another superpower of having synthetics you can be very flexible and make sure that the training data is representative And mirrors the true statistics you wanna you want to target. Okay, so let me show you how the data is actually Like synthesized on a high level what they do you have this template face You add the identity then you add certain expression like smiling or not Then you add the texture you add the hair clothing Sometimes they add like headwear eyewear what not like devices again That's something that Microsoft cares about like think Collins and finally you add like the environment You can see the lightning you can see the background etc So that's on a high level how these Like models are 3d avatars are produced and then you just render whatever you want by positioning a virtual camera in this scene So I think I want to stress here is this one. So it requires considerable expertise and Investment to develop a synthetics framework with minimal domain gap. So this is another reason I think this is not the way to go ahead, but it's a perfect perfect strategy for Microsoft in particular This is super expensive to build and maintain and etc So on a pro side say you have spent time labeling face images with landmarks However, you suddenly require additional landmarks in each image Relabeling and verifying will take a long long time if all of a sudden you realize you want to have additional landmark on the lip You have to literally take all of your images and send them back to labelers So that they can relabel the images and then you can actually Verify that you haven't introduced any mistakes by doing this so you can see this is very very time-consuming very very expensive and Erroneous, I guess Secondly, synthetics lets you render faces from a simulated device to develop algorithms and even guide hardware design itself Imagine what would be the the the alternative here So you literally have to design a prototype which is super expensive and slow process and then you'd have to create a prototype Process and then you'd have to collect data and only then can you train the model on the flip side with synthetics? You can literally just position in your in your in this in a scene You can just position a virtual camera and you can start rendering data training models and you can see whether it makes sense To position the sensor right there or not So this significantly reduces the iteration the time it takes you to iterate and so that's a huge huge asset So you may ask yourself. What is the thing with this framework? I mean we've already we We are already used to seeing amazingly realistic avatars in movies in in games etc And they give an answer to that here. So the visual effects industry has developed many techniques for convincing audiences That 3d faces are real and we build upon these in our approach However, a key difference is scale. So this framework proposes a procedural way to generate these like Avatars way more quickly and very realistically. So that's the value proposition of this whole thing And now let me kind of break it down for you how the how the models are created So we saw it on the on the high level here Now, let's see how they actually construct these identities expressions, etc. So the the main point here is the they basically collected a lot of these scans and To acquire these you either have to buy them or have the the space for collecting the scans yourself So the thing is those setups are very very pricey because I don't know if you ever saw one I'm gonna pop it somewhere on the screen here You basically have to have hundreds or if not thousands of cameras placed in that room in order to collect like the accurate geometry and everything you care about and later like Various processing algorithms to form these scans. Additionally, obviously you have to have real humans here in the loop So it's a it's a slow procedure. It's very expensive procedure. But once you have scans they additionally had to do the cleaning step So that's a very manual and very very very slow Like procedure to create from this raw scan to get this clean scan once you have the scans You can construct something called template face We're gonna see that in a bit then you can learn a generative face model and then you can sample from that face model to Obtain various geometry. Now, let's see a bit more about the generative face model itself, so you basically are trying to Create this generating function So the formula does look a bit intimidating and there is some domain expertise involved here But on a high level in order to form form a novel geometry with a certain expression What you do is you take this template face which is formed as and they mentioned here as the average face with neutral expressions So you take all of the scans you find their vertices you just do the average and you find the template head Then what they do is they learn this this basis and they call it the linear identity basis They learn that thing and finally you have this expression basis, which is not learned Okay once you have the template head once you have the this Linear identity basis and the expression basis you just use the betas in size to determine the actual geometry of the face And you can see here. So the generating function comprises out of Like takes as an input betas size and tatas so data just determines the position of the face model namely one of those is like jaw so you can basically position draw something like this and finally once they fit the so as I said this s is learned and Betas are also learned so that they fit the scans that they have in the data set Once they have a like a set of betas they can fit a multivariate like a normal distribution on top of it and that's how they actually form the the Genetic model and then you can just sample betas from it and you can get novel identities continuing on I want to stress again that this is a huge engineering project more so than than research and They say here that in addition to facial expression We'll layer random eye gaze directions on top of sampled expressions and use procedural logic to pose the eyelets accordingly So there is lots of heuristics all around the place here and not only for for the like positioning the the eyes But also here for texture you have to to you basically use the scans To get the texture information and they'll see here that they are modeling here at the strand level And so money here at the strand level allows us to capture realistic multi-path elimination effects, but on the con side Each asset was authored by a groom artist who specializes in creating digital hair So as you can see here, this is very slow and very very expensive Same goes for every other part of these of this synthetic framework So you have you have to obviously somebody has to construct these clothing somebody has to construct the hats the eyewear Etc but at the end because this is synthetics you can have various Various types of labels so not only vertices Which they later will see the results they they are using they are doing a landmark regression They'll you can also have UVs you can have masks you can have depth Normals albedo every single type of information you can care about in computer vision and graphics And again, this just showcased the variety of faces you can you can generate using using this the syntax framework And I think it's very very impressive Finally, let's jump to the actual results This is where they stress that this data-centric approach makes a lot of sense because for the semantic segmentation problem What I've done is used simplest off-the-shelf models such as unit and such as resnet to to to build Like to get results which are comparable with state-of-the-art methods on these benchmarks So for the semantics implementation what I do is they use unit a model and they just apply the binary cross entropy loss On the ground truth labels, which again are perfect because we have synthetic data And the second thing that has to be done here is something called label adaptation and let me first motivate what it is And then I'm gonna quickly explain how they do it So the thing is if you concentrate really hard on on on these images you can see that so this is the ground truth from the real data And you can see, uh, that the nose here It's fairly arbitrary decision for the labelers in this real data to to kind of end the nose here So basically this boundary of the nose is fairly arbitrarily chosen And so we shouldn't penalize synthetics for having chosen that the nose ends here instead of here and so that's why the adaptation adaptation is done and What what they do is they just freeze the weights of the first unit and they just what they do is they take the the real data Labels and they just kind of map using another unit So this is basically another unit and you just map from the output from this synthetics model You map to the real data. Let's jump to the final results Let's see the tables. So this is basically for the semantic segmentation problem You can see they compare their methods against various soda baselines and you can see results are fairly Comparable here. So 92 F1 score Against these baselines some of these baselines are a bit better. You can see that this paper in particular is 1.2 points better, but you need to realize that that model looks something like this So you basically have pyramid pooling here edge perceiving edge aware graph reasoning edge aware graph reasoning here Decoder fairly complex reprojection projection. So it's a fairly complex like architecture Like zooming in into this particular module you can see you have graph convolutions transpose bowling etc, etc So very very complex architectures. Whereas here they just use simple unit which is off-the-shelf model and they achieve comparable results So that's cool second thing is What confused me initially was the that we have better results with synthetics compared to real data And I guess the main reason here is this particular data set. So they used Helen data set I think it has around 2k training images. Whereas they have I think they used 100k images to train the unit. So that's why they get better results here Although it'd be really nice if they explicitly mentioned that somewhere that the amount of images used for this synthetics Like baseline synthetics results compared to the real result in here on the other hand on this different data set called LEPA The situation is a bit different because the real training data set actually has way more data points So the results here are better compared to synthetics data set and that makes sense. They also show results on the landmark localization And against some other Soda results and they show again that they have comparable results So the synthetics data again here some of them do have better results like like this one here or I don't know like this one Here etc. But like again, keep in mind they are using off-the-shelf models here It'd be very very interesting to see how this performs with much better models They did some ablations The augmentations matter a lot. So I forgot to mention that once you render our data What they do is they apply various augmentations like grayscaling blurring Warping etc. And they just show here that not using augmentations hurts performance by a lot You can see the jump here and finally Not doing the label adaptation hurts the performance a lot So you can see here it jumps to 5.61 where we want to be lower So it jumps from 3 to 5.6, which is a big gap So that means this part is very very important for landmark localization again The reason is the same there is also ambiguity same as with semantic segmentation and the nose example Here in the landmark localization problem They have one more interesting visualization here and they say here that predictions by network strain would reel So that's the top one against the synthetic data on the bottom one And they say that know how synthetic data network generalizes better across expression illumination pose and occlusion You can obviously see the results here are way better compared to the results up here It's maybe a knit but like claiming that you generalize better Whereas you have much more data using synthetics compared to real data is not a fair claim Although I guess it proves the point that it's better to have synthetic data because you can generate as much data as you want Yeah, and finally let's end up here I mentioned that the superpower is you can generate as much as dense landmarks as you want You can basically put virtual sensors wherever you want and render So for example eye tracking is something very important for the HoloLens device And so you need to be able to track where the pupil is where the eye is And you can you can quickly iterate with different positions of the cameras later when the prototype actually is developed You can be confident that the conclusions you draw from the synthetics experiments can now be Generalized to the real data and that's the superpower of this whole framework having said that they they clearly acknowledge certain limitations We do not include expression dependent wrinkling effects of realism suffers during certain expressions Since we sample parts of our model independently we sometimes get unusual but not impossible combinations such as feminine faces That have a beard and stuff like that. So yeah, I mean again, this is awesome result I think this for this particular problem that Microsoft and some of these companies like magic leap etc care about This is a perfect. I think this is a right direction to head into to build better products. But like if we're trying to just Talk here about improving research improving AI as a field. I think this is not definitely not the way to do to go ahead Finally, they mentioned that they are going to publish for non-commercial use case 100,000 images and I saw some people were confused on Twitter including Andrei Karpati And I thought maybe trying and giving an answer to that like question Namely to why they are only publishing 100k images and the answer lies pretty much here. I mean They took a look at the de-deblation for the landmark localization at task and they noticed that after a hundred key images Basically, they got diminishing returns and you can see the the mean error starts increasing again So that means a hundred case totally arbitrary value and the whole point of this framework is to have this Flexibility that I just explained here. So even when they render The all of these avatars from a certain angle. I mean the whole point is you want to be flexible enough to Put a new virtual camera and then we render the data there There is not much point in actually in my opinion in publishing these images So what would be really nice is if Microsoft were to make an API? Out of this and earn some money by doing that and help other people Like not have to go through the same procedure. That's as we saw was very expensive very slow to create this whole framework But I can also totally understand the reason for not doing that because it was super expensive and it's giving Microsoft an edge Anyways, if you like this paper straight out with a friend Subscribe to this channel join the discord communities. We have very smart people chatting there and you can ask them questions So yeah until next time. Bye. Bye
[{"start": 0.0, "end": 6.04, "text": " What's cracking guys in this video I'm covering this new paper called fake it till you make it face analysis in the wild using"}, {"start": 6.16, "end": 13.9, "text": " synthetic data alone by these awesome people from Microsoft and I actually had the privilege of working with them while I was back at"}, {"start": 13.9, "end": 15.66, "text": " Microsoft so as a quick disclaimer"}, {"start": 15.66, "end": 19.82, "text": " I actually did contribute a little bit to this framework the synthetics framework"}, {"start": 19.82, "end": 24.54, "text": " And so I know a lot of details of how this project was structured and everything in this video"}, {"start": 24.54, "end": 29.900000000000002, "text": " I'm gonna only tell you obviously only the things and conclusions you can you can deduce from the paper itself"}, {"start": 29.9, "end": 35.58, "text": " So having said that these guys are a real wood Tata spell through shade this Charlie Hewitt Sebastian Gaggio"}, {"start": 36.04, "end": 41.879999999999995, "text": " Matthew Johnson Virginia as tellers Thomas Cashman and Jamie Sutton, so let's see what the hype is all about"}, {"start": 42.58, "end": 50.0, "text": " They say here that we demonstrated it's possible to perform face related computer vision in the wild using synthetic data alone"}, {"start": 50.239999999999995, "end": 55.84, "text": " So I mean obviously you want to use synthetic see if that's possible because you have perfect ground truth"}, {"start": 55.84, "end": 58.08, "text": " But what was the problem so far well?"}, {"start": 58.08, "end": 64.46, "text": " there is something called domain gap and I think it's fairly obvious when you try and train your models on synthetic data and"}, {"start": 64.8, "end": 70.28, "text": " Then try to do to do inference with that same model on real data. You're gonna have some problems"}, {"start": 70.6, "end": 76.25999999999999, "text": " The reason is you have different distributions in the case where your synthetics is not perfect as it usually is not"}, {"start": 77.0, "end": 79.16, "text": " you basically have a gap because the"}, {"start": 79.68, "end": 85.42, "text": " distribution between the those data points from the synthetics is different from the distribution of the data points in the real data and"}, {"start": 85.42, "end": 93.38, "text": " So different people different groups try to solve this in different ways and some good examples would be like opening eye"}, {"start": 93.62, "end": 95.5, "text": " so I"}, {"start": 95.5, "end": 102.98, "text": " Recently a couple months ago covered this opening ice paper where they used the robotic hand to solve the Rubik's Cube and the thing"}, {"start": 102.98, "end": 107.32000000000001, "text": " Is the policy the RL agent was actually trained purely in simulation?"}, {"start": 107.34, "end": 112.98, "text": " The approach they took was to not to try to create a perfect simulation because it's even harder"}, {"start": 112.98, "end": 117.5, "text": " To create a perfect simulator for this robotic hand then to create a simulator"}, {"start": 118.02000000000001, "end": 122.86, "text": " For the for the faces because here you only need to mimic the appearance of these faces"}, {"start": 122.94, "end": 129.18, "text": " Whereas in the in this case you have to model the physics you have to model the friction you have to model the contact"}, {"start": 129.18, "end": 132.38, "text": " Forces which is super hard to do and so what they did is just"}, {"start": 133.3, "end": 139.38, "text": " Randomize the simulation so that the real world appears like just another instantiation of a simulation"}, {"start": 139.38, "end": 142.7, "text": " So they modify even stuff like gravity from the usual"}, {"start": 143.22, "end": 148.62, "text": " 9.81 at least on earth to like even 8.5 or whatnot, but it was all done gradually"}, {"start": 149.18, "end": 152.9, "text": " In order to have this curriculum like like a gradual training"}, {"start": 152.9, "end": 158.18, "text": " So that was one approach second approach other approaches were stuff like you can fine-tune"}, {"start": 158.57999999999998, "end": 165.14, "text": " On real images after training on your synthetics, so you have imperfect synthetics you fine-tune in real data from the target distribution"}, {"start": 165.14, "end": 170.26, "text": " And then you can expect that the the accuracy or whatever metric you care about is going to improve"}, {"start": 170.73999999999998, "end": 177.7, "text": " And finally there are approaches such as uprealing with GANs whereby you take your real data, and you pass them through certain like"}, {"start": 178.26, "end": 182.1, "text": " GANs which will make those images appear more realistic"}, {"start": 182.1, "end": 186.1, "text": " Those are all the different ways to try to reduce this domain gap so"}, {"start": 186.42, "end": 191.38, "text": " Microsoft on the other hand took another route and that's to reduce the domain gap almost down to zero"}, {"start": 191.38, "end": 195.78, "text": " And we have to have some bad jokes on this on this channel. So yeah"}, {"start": 196.42, "end": 197.7, "text": " anyways"}, {"start": 197.7, "end": 204.18, "text": " Basically, here are some of the images that the framework produces you can see they are fairly realistic"}, {"start": 205.06, "end": 210.18, "text": " Especially this this this this person here. You can see that you can have geometry"}, {"start": 210.18, "end": 216.5, "text": " So 3d information because you have a synthetic data and that means you also have perfect labeling, which is awesome"}, {"start": 216.5, "end": 222.74, "text": " Which is the superpower of this whole framework. You can do pixel wise like a semantic segmentation"}, {"start": 222.98, "end": 227.46, "text": " You have the landmarks however dense you want them to be and a lot of other"}, {"start": 228.02, "end": 231.62, "text": " Ground-truth data which we're going to see a bit later and before I dig deeper into the paper"}, {"start": 231.7, "end": 237.94, "text": " Let me just show you this quick clip from Microsoft where you'll see the animations of the of this of this syntax framework"}, {"start": 237.94, "end": 245.94, "text": " Here are some examples synthetic faces that we render as training data for machine learning"}, {"start": 246.34, "end": 250.74, "text": " With synthetic data you can guarantee perfect labels without annotation noise"}, {"start": 251.14, "end": 259.06, "text": " Generate rich labels that are otherwise impossible to label by hand and have full control over variation and diversity in a data set"}, {"start": 261.38, "end": 264.9, "text": " Awesome, so let's continue and dig deeper into this paper"}, {"start": 264.9, "end": 270.58, "text": " So they say here that they show that the synthetic data can match can both match real data inaccuracy as well as open up"}, {"start": 270.58, "end": 276.82, "text": " New approaches where manual labeling would be impossible. Okay, so a quick remark here. You need to keep in mind that"}, {"start": 277.76, "end": 278.82, "text": " rendering"}, {"start": 278.82, "end": 284.58, "text": " Only face images so the heads of of humans is a fairly constrained problem"}, {"start": 284.58, "end": 285.7, "text": " I mean you have"}, {"start": 285.7, "end": 290.17999999999995, "text": " Uh, like a certain geometry that all humans have we have usually have two eyes"}, {"start": 290.18, "end": 296.34000000000003, "text": " We usually have one nose and I mean the problem is very very constrained and I really believe that we can"}, {"start": 296.82, "end": 301.54, "text": " Actually reduce to the main gap almost down to zero if not to zero in the future"}, {"start": 302.1, "end": 309.06, "text": " For this particular problem that Microsoft deeply cares about so because this the team works in mixed reality and virtual reality"}, {"start": 309.22, "end": 313.7, "text": " so that means they they care about stuff like Microsoft HoloLens and"}, {"start": 314.26, "end": 317.38, "text": " There are other competitors out there such as Magic Leap and Facebook"}, {"start": 317.38, "end": 323.62, "text": " Reality Labs and all of them care about this very constrained problem of like understanding"}, {"start": 323.86, "end": 326.18, "text": " human faces and bodies and so"}, {"start": 327.62, "end": 332.9, "text": " That the point I'm trying to drive here is that if you had some other area such as like self-driving"}, {"start": 333.06, "end": 340.1, "text": " I'm fairly sure this is not the way I had because I mean it boils down to the question whether it's easier to"}, {"start": 340.9, "end": 343.42, "text": " build up a new world or just understand the"}, {"start": 343.42, "end": 349.14000000000004, "text": " The the the current world because in order to build a perfect simulator for self-driving you literally have to"}, {"start": 349.54, "end": 355.86, "text": " Cover so many edge cases, etc. So you literally have to build something like I guess real-world simulator and"}, {"start": 356.70000000000005, "end": 364.66, "text": " Yeah, I'm fairly sure that's harder than just trying to build a computer vision systems that will solve us self-driving continuing on here"}, {"start": 365.54, "end": 371.98, "text": " When they say that when faced with a machine learning problem the hardest challenge often isn't choosing the right machine learning model"}, {"start": 371.98, "end": 377.18, "text": " It's finding the right data and this reminds me strongly of Andrew Eng's data centric approach"}, {"start": 377.58000000000004, "end": 384.40000000000003, "text": " And the thing I noticed here is that all of these people and groups that are focused more on short-term products"}, {"start": 385.90000000000003, "end": 392.06, "text": " Tend to say something like this and I don't mean that in a in a in a bad way. I just think that"}, {"start": 392.94, "end": 397.26, "text": " like this implies obviously implies supervised learning paradigm and"}, {"start": 397.26, "end": 404.7, "text": " I'm fairly sure that's not the way that kids and humans learn and I'm fairly sure that approaches that for example Facebook and deep minor pushing"}, {"start": 404.7, "end": 410.98, "text": " Like self supervised paradigm is the way to go long term. But having said that again, I"}, {"start": 411.74, "end": 414.44, "text": " Think this is a fairly fairly good approach"}, {"start": 414.86, "end": 419.65999999999997, "text": " At the time being because we're gonna build better product which will enable us accelerate"}, {"start": 420.09999999999997, "end": 422.82, "text": " Accomplishing those long-term goals of cracking intelligence"}, {"start": 422.82, "end": 426.21999999999997, "text": " Obviously they mentioned some some benefits of synthetic data here"}, {"start": 426.21999999999997, "end": 430.7, "text": " So with synthetic data you can guarantee perfect labels without annotation noise"}, {"start": 431.26, "end": 433.78, "text": " Generate rich labels, which are otherwise"}, {"start": 434.38, "end": 439.21999999999997, "text": " Impossible to label by hand and they have an example later here in the paper. Let me show you"}, {"start": 440.1, "end": 442.5, "text": " basically, I mean you can you can"}, {"start": 443.21999999999997, "end": 444.42, "text": " generate"}, {"start": 444.42, "end": 451.1, "text": " As much landmarks as you want like you can you can make the the mesh as dense as you want and this is obviously not possible"}, {"start": 451.1, "end": 456.42, "text": " To achieve by by manual labeling it would be super expensive and super super error prone"}, {"start": 456.74, "end": 462.82000000000005, "text": " So that's some of the benefits that real data that synthetic data is bringing. Let me go back here"}, {"start": 463.46000000000004, "end": 468.14000000000004, "text": " Aside from that you have a full control over variation and diversity in a data set"}, {"start": 468.14000000000004, "end": 472.06, "text": " Which is also very very important when we are deploying these products globally"}, {"start": 472.06, "end": 477.22, "text": " You want to make sure it works on different demographics and they even have a histogram down here"}, {"start": 477.22, "end": 481.5, "text": " Let me quickly show you this so they show the histograms here over the age"}, {"start": 481.98, "end": 487.82000000000005, "text": " Over the I guess it's sex not gender and and ethnicity and as you can see here"}, {"start": 487.82000000000005, "end": 492.5, "text": " It's it's a work in progress obviously, but the good thing is you can basically improve this"}, {"start": 492.5, "end": 496.86, "text": " Whereas it's way harder to do this when you're trying to collect real data"}, {"start": 497.46000000000004, "end": 498.66, "text": " and"}, {"start": 498.66, "end": 503.78000000000003, "text": " I guess a small remark here. It'd be nice to have like a ground truth histograms here"}, {"start": 503.78, "end": 509.5, "text": " I the global statistics so that we can compare how far away is this current?"}, {"start": 510.02, "end": 515.54, "text": " Histogram from the actual desired target histogram. So this is another superpower of having synthetics"}, {"start": 515.54, "end": 520.06, "text": " you can be very flexible and make sure that the training data is"}, {"start": 521.06, "end": 522.06, "text": " representative"}, {"start": 522.06, "end": 529.14, "text": " And mirrors the true statistics you wanna you want to target. Okay, so let me show you how the data is actually"}, {"start": 529.14, "end": 533.34, "text": " Like synthesized on a high level what they do you have this template face"}, {"start": 533.86, "end": 538.54, "text": " You add the identity then you add certain expression like smiling or not"}, {"start": 538.54, "end": 541.18, "text": " Then you add the texture you add the hair clothing"}, {"start": 541.58, "end": 545.98, "text": " Sometimes they add like headwear eyewear what not like devices again"}, {"start": 545.98, "end": 551.9399999999999, "text": " That's something that Microsoft cares about like think Collins and finally you add like the environment"}, {"start": 551.9399999999999, "end": 553.98, "text": " You can see the lightning you can see the background etc"}, {"start": 553.98, "end": 555.98, "text": " So that's on a high level how these"}, {"start": 555.98, "end": 563.98, "text": " Like models are 3d avatars are produced and then you just render whatever you want by positioning a virtual camera in this scene"}, {"start": 563.98, "end": 569.98, "text": " So I think I want to stress here is this one. So it requires considerable expertise and"}, {"start": 570.58, "end": 575.98, "text": " Investment to develop a synthetics framework with minimal domain gap. So this is another reason"}, {"start": 575.98, "end": 581.98, "text": " I think this is not the way to go ahead, but it's a perfect perfect strategy for Microsoft in particular"}, {"start": 581.98, "end": 586.98, "text": " This is super expensive to build and maintain and etc"}, {"start": 586.98, "end": 591.98, "text": " So on a pro side say you have spent time labeling face images with landmarks"}, {"start": 591.98, "end": 595.98, "text": " However, you suddenly require additional landmarks in each image"}, {"start": 595.98, "end": 601.98, "text": " Relabeling and verifying will take a long long time if all of a sudden you realize you want to have additional landmark on the lip"}, {"start": 601.98, "end": 605.98, "text": " You have to literally take all of your images and send them back to labelers"}, {"start": 605.98, "end": 609.98, "text": " So that they can relabel the images and then you can actually"}, {"start": 609.98, "end": 616.98, "text": " Verify that you haven't introduced any mistakes by doing this so you can see this is very very time-consuming very very expensive"}, {"start": 616.98, "end": 618.98, "text": " and"}, {"start": 618.98, "end": 620.98, "text": " Erroneous, I guess"}, {"start": 620.98, "end": 627.98, "text": " Secondly, synthetics lets you render faces from a simulated device to develop algorithms and even guide hardware design itself"}, {"start": 627.98, "end": 630.98, "text": " Imagine what would be the the the alternative here"}, {"start": 630.98, "end": 637.98, "text": " So you literally have to design a prototype which is super expensive and slow process and then you'd have to create a prototype"}, {"start": 637.98, "end": 644.98, "text": " Process and then you'd have to collect data and only then can you train the model on the flip side with synthetics?"}, {"start": 644.98, "end": 649.26, "text": " You can literally just position in your in your in this in a scene"}, {"start": 649.26, "end": 655.7, "text": " You can just position a virtual camera and you can start rendering data training models and you can see whether it makes sense"}, {"start": 655.7, "end": 658.1800000000001, "text": " To position the sensor right there or not"}, {"start": 658.1800000000001, "end": 664.9, "text": " So this significantly reduces the iteration the time it takes you to iterate and so that's a huge huge asset"}, {"start": 664.9, "end": 670.98, "text": " So you may ask yourself. What is the thing with this framework? I mean we've already we"}, {"start": 671.5799999999999, "end": 673.5799999999999, "text": " We are already used to seeing"}, {"start": 674.1, "end": 678.4599999999999, "text": " amazingly realistic avatars in movies in in games etc"}, {"start": 678.4599999999999, "end": 685.38, "text": " And they give an answer to that here. So the visual effects industry has developed many techniques for convincing audiences"}, {"start": 686.02, "end": 690.02, "text": " That 3d faces are real and we build upon these in our approach"}, {"start": 690.02, "end": 696.92, "text": " However, a key difference is scale. So this framework proposes a procedural way to generate these like"}, {"start": 697.66, "end": 703.06, "text": " Avatars way more quickly and very realistically. So that's the value proposition of this whole thing"}, {"start": 703.06, "end": 707.8, "text": " And now let me kind of break it down for you how the how the models are created"}, {"start": 707.8, "end": 709.8, "text": " So we saw it on the on the high level here"}, {"start": 710.38, "end": 718.1999999999999, "text": " Now, let's see how they actually construct these identities expressions, etc. So the the main point here is the they basically"}, {"start": 718.2, "end": 720.2, "text": " collected a"}, {"start": 720.44, "end": 722.44, "text": " lot of these scans and"}, {"start": 722.8000000000001, "end": 728.24, "text": " To acquire these you either have to buy them or have the the space for collecting the scans yourself"}, {"start": 728.24, "end": 732.44, "text": " So the thing is those setups are very very pricey because I don't know if you ever saw one"}, {"start": 732.44, "end": 734.6800000000001, "text": " I'm gonna pop it somewhere on the screen here"}, {"start": 734.6800000000001, "end": 737.9200000000001, "text": " You basically have to have hundreds or if not thousands of cameras"}, {"start": 738.4000000000001, "end": 743.9200000000001, "text": " placed in that room in order to collect like the accurate geometry and everything you care about and"}, {"start": 744.5600000000001, "end": 746.5600000000001, "text": " later like"}, {"start": 746.56, "end": 752.0799999999999, "text": " Various processing algorithms to form these scans. Additionally, obviously you have to have real humans here in the loop"}, {"start": 752.0799999999999, "end": 759.16, "text": " So it's a it's a slow procedure. It's very expensive procedure. But once you have scans they additionally had to do the cleaning step"}, {"start": 759.16, "end": 762.3199999999999, "text": " So that's a very manual and very very very slow"}, {"start": 762.52, "end": 767.7199999999999, "text": " Like procedure to create from this raw scan to get this clean scan once you have the scans"}, {"start": 767.7199999999999, "end": 769.8, "text": " You can construct something called template face"}, {"start": 769.8, "end": 775.2399999999999, "text": " We're gonna see that in a bit then you can learn a generative face model and then you can sample from that face model to"}, {"start": 775.24, "end": 780.08, "text": " Obtain various geometry. Now, let's see a bit more about the generative face model"}, {"start": 780.76, "end": 784.12, "text": " itself, so you basically are trying to"}, {"start": 785.8, "end": 787.64, "text": " Create this generating function"}, {"start": 787.64, "end": 791.84, "text": " So the formula does look a bit intimidating and there is some domain expertise involved here"}, {"start": 791.84, "end": 797.32, "text": " But on a high level in order to form form a novel geometry with a certain expression"}, {"start": 797.32, "end": 804.76, "text": " What you do is you take this template face which is formed as and they mentioned here as the average face with neutral expressions"}, {"start": 804.76, "end": 810.4, "text": " So you take all of the scans you find their vertices you just do the average and you find the template head"}, {"start": 810.76, "end": 817.2, "text": " Then what they do is they learn this this basis and they call it the linear identity basis"}, {"start": 817.68, "end": 821.68, "text": " They learn that thing and finally you have this expression basis, which is not learned"}, {"start": 821.92, "end": 822.4399999999999, "text": " Okay"}, {"start": 822.4399999999999, "end": 825.04, "text": " once you have the template head once you have the this"}, {"start": 825.4, "end": 833.28, "text": " Linear identity basis and the expression basis you just use the betas in size to determine the actual geometry of the face"}, {"start": 833.28, "end": 837.04, "text": " And you can see here. So the generating function comprises out of"}, {"start": 837.56, "end": 843.68, "text": " Like takes as an input betas size and tatas so data just determines the position of the face model"}, {"start": 843.8399999999999, "end": 848.48, "text": " namely one of those is like jaw so you can basically position draw something like this and"}, {"start": 849.64, "end": 853.8399999999999, "text": " finally once they fit the so as I said this s is learned and"}, {"start": 854.24, "end": 859.24, "text": " Betas are also learned so that they fit the scans that they have in the data set"}, {"start": 859.24, "end": 865.6800000000001, "text": " Once they have a like a set of betas they can fit a multivariate like a normal distribution on top of it"}, {"start": 865.6800000000001, "end": 868.36, "text": " and that's how they actually form the the"}, {"start": 869.48, "end": 874.0, "text": " Genetic model and then you can just sample betas from it and you can get novel identities"}, {"start": 874.6, "end": 882.76, "text": " continuing on I want to stress again that this is a huge engineering project more so than than research and"}, {"start": 883.48, "end": 885.48, "text": " They say here that in addition to facial expression"}, {"start": 885.48, "end": 892.9200000000001, "text": " We'll layer random eye gaze directions on top of sampled expressions and use procedural logic to pose the eyelets accordingly"}, {"start": 892.9200000000001, "end": 900.5600000000001, "text": " So there is lots of heuristics all around the place here and not only for for the like positioning the the eyes"}, {"start": 900.5600000000001, "end": 905.72, "text": " But also here for texture you have to to you basically use the scans"}, {"start": 906.2, "end": 911.36, "text": " To get the texture information and they'll see here that they are modeling here at the strand level"}, {"start": 911.36, "end": 917.64, "text": " And so money here at the strand level allows us to capture realistic multi-path elimination effects, but on the con side"}, {"start": 918.12, "end": 923.04, "text": " Each asset was authored by a groom artist who specializes in creating digital hair"}, {"start": 923.08, "end": 927.0, "text": " So as you can see here, this is very slow and very very expensive"}, {"start": 927.08, "end": 930.92, "text": " Same goes for every other part of these of this synthetic framework"}, {"start": 930.92, "end": 938.84, "text": " So you have you have to obviously somebody has to construct these clothing somebody has to construct the hats the eyewear"}, {"start": 938.84, "end": 942.76, "text": " Etc but at the end because this is synthetics you can have various"}, {"start": 943.5600000000001, "end": 947.2800000000001, "text": " Various types of labels so not only vertices"}, {"start": 947.8000000000001, "end": 953.0, "text": " Which they later will see the results they they are using they are doing a landmark regression"}, {"start": 953.0, "end": 955.6800000000001, "text": " They'll you can also have UVs you can have masks you can have depth"}, {"start": 956.12, "end": 961.48, "text": " Normals albedo every single type of information you can care about in computer vision and graphics"}, {"start": 961.48, "end": 970.04, "text": " And again, this just showcased the variety of faces you can you can generate using using this the syntax framework"}, {"start": 970.04, "end": 972.04, "text": " And I think it's very very impressive"}, {"start": 972.52, "end": 974.52, "text": " Finally, let's jump to the actual results"}, {"start": 975.32, "end": 981.8000000000001, "text": " This is where they stress that this data-centric approach makes a lot of sense because for the semantic segmentation problem"}, {"start": 981.8000000000001, "end": 989.04, "text": " What I've done is used simplest off-the-shelf models such as unit and such as resnet to to to build"}, {"start": 989.04, "end": 993.92, "text": " Like to get results which are comparable with state-of-the-art methods on these benchmarks"}, {"start": 994.0799999999999, "end": 1001.1999999999999, "text": " So for the semantics implementation what I do is they use unit a model and they just apply the binary cross entropy loss"}, {"start": 1001.4399999999999, "end": 1005.9599999999999, "text": " On the ground truth labels, which again are perfect because we have synthetic data"}, {"start": 1005.9599999999999, "end": 1012.88, "text": " And the second thing that has to be done here is something called label adaptation and let me first motivate what it is"}, {"start": 1012.88, "end": 1014.78, "text": " And then I'm gonna quickly explain how they do it"}, {"start": 1014.78, "end": 1022.16, "text": " So the thing is if you concentrate really hard on on on these images you can see that so this is the ground truth from the real data"}, {"start": 1022.56, "end": 1025.2, "text": " And you can see, uh, that the nose here"}, {"start": 1025.6, "end": 1031.84, "text": " It's fairly arbitrary decision for the labelers in this real data to to kind of end the nose here"}, {"start": 1032.0, "end": 1035.68, "text": " So basically this boundary of the nose is fairly arbitrarily chosen"}, {"start": 1036.0, "end": 1041.52, "text": " And so we shouldn't penalize synthetics for having chosen that the nose ends here instead of here"}, {"start": 1041.52, "end": 1044.48, "text": " and so that's why the adaptation adaptation is done and"}, {"start": 1045.36, "end": 1053.12, "text": " What what they do is they just freeze the weights of the first unit and they just what they do is they take the the real data"}, {"start": 1053.48, "end": 1056.8, "text": " Labels and they just kind of map using another unit"}, {"start": 1056.8, "end": 1063.44, "text": " So this is basically another unit and you just map from the output from this synthetics model"}, {"start": 1063.8, "end": 1066.8, "text": " You map to the real data. Let's jump to the final results"}, {"start": 1066.8, "end": 1072.28, "text": " Let's see the tables. So this is basically for the semantic segmentation problem"}, {"start": 1072.28, "end": 1080.36, "text": " You can see they compare their methods against various soda baselines and you can see results are fairly"}, {"start": 1080.6399999999999, "end": 1084.58, "text": " Comparable here. So 92 F1 score"}, {"start": 1085.1, "end": 1090.84, "text": " Against these baselines some of these baselines are a bit better. You can see that this paper in particular is"}, {"start": 1090.84, "end": 1096.6399999999999, "text": " 1.2 points better, but you need to realize that that model looks something like this"}, {"start": 1096.6399999999999, "end": 1104.52, "text": " So you basically have pyramid pooling here edge perceiving edge aware graph reasoning edge aware graph reasoning here"}, {"start": 1104.84, "end": 1110.54, "text": " Decoder fairly complex reprojection projection. So it's a fairly complex like architecture"}, {"start": 1111.0, "end": 1118.1599999999999, "text": " Like zooming in into this particular module you can see you have graph convolutions transpose bowling etc, etc"}, {"start": 1118.16, "end": 1125.4, "text": " So very very complex architectures. Whereas here they just use simple unit which is off-the-shelf model and they achieve comparable results"}, {"start": 1125.4, "end": 1126.72, "text": " So that's cool"}, {"start": 1126.72, "end": 1128.24, "text": " second thing is"}, {"start": 1128.24, "end": 1133.6000000000001, "text": " What confused me initially was the that we have better results with synthetics compared to real data"}, {"start": 1133.6000000000001, "end": 1139.0400000000002, "text": " And I guess the main reason here is this particular data set. So they used Helen data set"}, {"start": 1139.0400000000002, "end": 1143.16, "text": " I think it has around 2k training images. Whereas they have I think they used"}, {"start": 1143.16, "end": 1147.6000000000001, "text": " 100k images to train the unit. So that's why they get better results here"}, {"start": 1147.6000000000001, "end": 1154.4, "text": " Although it'd be really nice if they explicitly mentioned that somewhere that the amount of images used for this synthetics"}, {"start": 1155.0400000000002, "end": 1162.3600000000001, "text": " Like baseline synthetics results compared to the real result in here on the other hand on this different data set called LEPA"}, {"start": 1163.44, "end": 1168.5600000000002, "text": " The situation is a bit different because the real training data set actually has way more data points"}, {"start": 1168.56, "end": 1175.12, "text": " So the results here are better compared to synthetics data set and that makes sense. They also show results on the landmark localization"}, {"start": 1175.44, "end": 1177.44, "text": " And against some other"}, {"start": 1177.8799999999999, "end": 1181.7, "text": " Soda results and they show again that they have comparable results"}, {"start": 1181.72, "end": 1188.8, "text": " So the synthetics data again here some of them do have better results like like this one here or I don't know like this one"}, {"start": 1188.8, "end": 1193.44, "text": " Here etc. But like again, keep in mind they are using off-the-shelf models here"}, {"start": 1193.44, "end": 1197.74, "text": " It'd be very very interesting to see how this performs with much better models"}, {"start": 1197.74, "end": 1199.74, "text": " They did some ablations"}, {"start": 1200.82, "end": 1205.34, "text": " The augmentations matter a lot. So I forgot to mention that once you render our data"}, {"start": 1205.94, "end": 1209.86, "text": " What they do is they apply various augmentations like grayscaling blurring"}, {"start": 1210.78, "end": 1216.9, "text": " Warping etc. And they just show here that not using augmentations hurts performance by a lot"}, {"start": 1216.9, "end": 1220.18, "text": " You can see the jump here and finally"}, {"start": 1220.98, "end": 1224.22, "text": " Not doing the label adaptation hurts the performance a lot"}, {"start": 1224.22, "end": 1227.9, "text": " So you can see here it jumps to 5.61 where we want to be lower"}, {"start": 1227.9, "end": 1231.9, "text": " So it jumps from 3 to 5.6, which is a big gap"}, {"start": 1231.9, "end": 1236.46, "text": " So that means this part is very very important for landmark localization again"}, {"start": 1236.46, "end": 1241.84, "text": " The reason is the same there is also ambiguity same as with semantic segmentation and the nose example"}, {"start": 1242.34, "end": 1244.66, "text": " Here in the landmark localization problem"}, {"start": 1244.8600000000001, "end": 1250.46, "text": " They have one more interesting visualization here and they say here that predictions by network strain would reel"}, {"start": 1250.46, "end": 1255.3, "text": " So that's the top one against the synthetic data on the bottom one"}, {"start": 1255.3, "end": 1261.9, "text": " And they say that know how synthetic data network generalizes better across expression illumination pose and occlusion"}, {"start": 1261.9, "end": 1265.74, "text": " You can obviously see the results here are way better compared to the results up here"}, {"start": 1265.74, "end": 1268.38, "text": " It's maybe a knit but like claiming that you generalize better"}, {"start": 1268.78, "end": 1273.82, "text": " Whereas you have much more data using synthetics compared to real data is not a fair claim"}, {"start": 1273.82, "end": 1279.66, "text": " Although I guess it proves the point that it's better to have synthetic data because you can generate as much data as you want"}, {"start": 1279.66, "end": 1284.14, "text": " Yeah, and finally let's end up here"}, {"start": 1284.14, "end": 1291.5800000000002, "text": " I mentioned that the superpower is you can generate as much as dense landmarks as you want"}, {"start": 1291.5800000000002, "end": 1296.6200000000001, "text": " You can basically put virtual sensors wherever you want and render"}, {"start": 1296.6200000000001, "end": 1300.3000000000002, "text": " So for example eye tracking is something very important for the HoloLens device"}, {"start": 1300.3000000000002, "end": 1305.1000000000001, "text": " And so you need to be able to track where the pupil is where the eye is"}, {"start": 1305.1, "end": 1312.06, "text": " And you can you can quickly iterate with different positions of the cameras later when the prototype actually is developed"}, {"start": 1312.06, "end": 1317.6599999999999, "text": " You can be confident that the conclusions you draw from the synthetics experiments can now be"}, {"start": 1318.1399999999999, "end": 1325.62, "text": " Generalized to the real data and that's the superpower of this whole framework having said that they they clearly acknowledge certain limitations"}, {"start": 1325.62, "end": 1330.8999999999999, "text": " We do not include expression dependent wrinkling effects of realism suffers during certain expressions"}, {"start": 1330.9, "end": 1338.22, "text": " Since we sample parts of our model independently we sometimes get unusual but not impossible combinations such as feminine faces"}, {"start": 1338.22, "end": 1345.38, "text": " That have a beard and stuff like that. So yeah, I mean again, this is awesome result"}, {"start": 1345.38, "end": 1351.5, "text": " I think this for this particular problem that Microsoft and some of these companies like magic leap etc care about"}, {"start": 1351.98, "end": 1358.5400000000002, "text": " This is a perfect. I think this is a right direction to head into to build better products. But like if we're trying to just"}, {"start": 1358.54, "end": 1365.34, "text": " Talk here about improving research improving AI as a field. I think this is not definitely not the way to do to go ahead"}, {"start": 1365.34, "end": 1370.62, "text": " Finally, they mentioned that they are going to publish for non-commercial use case"}, {"start": 1370.62, "end": 1375.8999999999999, "text": " 100,000 images and I saw some people were confused on Twitter including Andrei Karpati"}, {"start": 1375.8999999999999, "end": 1380.22, "text": " And I thought maybe trying and giving an answer to that like question"}, {"start": 1380.22, "end": 1386.46, "text": " Namely to why they are only publishing 100k images and the answer lies pretty much here. I mean"}, {"start": 1386.46, "end": 1394.3400000000001, "text": " They took a look at the de-deblation for the landmark localization at task and they noticed that after a hundred key images"}, {"start": 1395.1000000000001, "end": 1400.42, "text": " Basically, they got diminishing returns and you can see the the mean error starts increasing again"}, {"start": 1400.42, "end": 1406.98, "text": " So that means a hundred case totally arbitrary value and the whole point of this framework is to have this"}, {"start": 1407.46, "end": 1411.92, "text": " Flexibility that I just explained here. So even when they render"}, {"start": 1411.92, "end": 1418.48, "text": " The all of these avatars from a certain angle. I mean the whole point is you want to be flexible enough to"}, {"start": 1419.2, "end": 1422.2, "text": " Put a new virtual camera and then we render the data there"}, {"start": 1422.2, "end": 1426.8600000000001, "text": " There is not much point in actually in my opinion in publishing these images"}, {"start": 1426.8600000000001, "end": 1430.88, "text": " So what would be really nice is if Microsoft were to make an API?"}, {"start": 1431.48, "end": 1436.04, "text": " Out of this and earn some money by doing that and help other people"}, {"start": 1436.04, "end": 1443.04, "text": " Like not have to go through the same procedure. That's as we saw was very expensive very slow to create this whole framework"}, {"start": 1443.04, "end": 1450.12, "text": " But I can also totally understand the reason for not doing that because it was super expensive and it's giving Microsoft an edge"}, {"start": 1450.24, "end": 1452.86, "text": " Anyways, if you like this paper straight out with a friend"}, {"start": 1453.8, "end": 1459.1599999999999, "text": " Subscribe to this channel join the discord communities. We have very smart people chatting there and you can ask them questions"}, {"start": 1459.16, "end": 1465.96, "text": " So yeah until next time. Bye. Bye"}]