Efficient Geometry-aware 3D Generative Adversarial Networks | GAN Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover the "Efficient Geometry-aware 3D Generative Adversarial Networks" paper, that introduced a novel explicit-implicit 3D scene representation network and devised a novel framework (dual discrimination, gen/disc pose conditioning, etc.) for SOTA 3D GANs.

I also explain how NeRF works in case you didn't have to chance to read it, yet.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ This video paper: https://arxiv.org/abs/2112.07945
✅ Website: https://matthew-a-chan.github.io/EG3D/

Supporting papers:
✅ NeRF paper: https://arxiv.org/abs/2003.08934
✅ Explicit voxel grids paper: https://arxiv.org/abs/1906.07751
✅ StyleGAN v1: https://arxiv.org/abs/1812.04948
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:45 Tri-plane 3D scene representation intro
04:25 NeRF in depth
13:35 Explicit voxel grid methods
14:40 Tri-plane 3D scene representation explained
16:40 Tri-plane is as expressive
18:30 Efficient 3D GANs pipeline
22:05 Pose correlated facial features
23:40 Dual discrimination, super-resolution
32:40 Results
37:00 Ethical considerations, are we going to change anything?
39:25 Intrinsics, extrinsics robustness?
42:00 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Kulsoom Abdullah
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#3D #GANs #NeRF
 What's cracking guys? In this video I'm going to cover a very interesting paper and as a quick note I still don't have as you can see I don't have the video and the YouTube studio set up yet So I'm gonna have to only show you the paper for the time being hopefully I'm going to solve that very quickly But basically I'm covering this novel very interesting paper called efficient geometry aware 3d generative adversarial networks or GANs for short from this beautiful group of people from Stanford and Nvidia as a quick knit I think that like the very fact that this is 3d I mean, I think this is a pleonasm because 3d should imply the geometry aware part of this title But yeah in any case I guess it makes makes a more interesting title okay before I start digging into the actual paper Let me show you their video. We present a new 3d GAN that produces photorealistic images that are both multi-view consistent and Geometry aware Our approach enables unsupervised learning of high fidelity 3d representations without any explicit 3d or multi-view supervision So you can see results are fairly impressive because they are they are multi-view consistent You can see that even when you're changing the poses the appearance and the identity Remain the same which is kind of tricky for GANs to achieve And you can see that so here what's interesting is you can see let me stop it here you saw that the texture is kind of sticking and So when the cat is moving when the pose is changing you can see that the fur is kind of flickering or something So that stickiness problem. I think it's called like that like texture sticking problem was solved in one of the Newer style GAN papers if I'm not wrong Okay, in any case let's get back to the paper. We saw the the visualizations there and Now let me try and explain why this paper is special and very interesting so First thing they say here is that they train this whole thing this GAN without any ground truth 3d scans or Multi-view supervision so that means they just have 2d images basically and they manage to achieve this this Like basically multi-view consistency as you can see here the pose here changes But the identity remains the same and they also managed to recover the underlying 3d geometry Of the scene which is very cool now in order to understand how this Like paper works. We have to start with this novel Explicit implicit representation they devised called a tree plane. I think it's called tree plane representation or something but basically it's a combination it's a Somewhere in the middle between nerf, and I'm gonna briefly I'm gonna shortly explain what nerf exactly is and these like explicit voxel approaches And when you combine the good sides of this approach and the nerf approach you get this novel representation That they've devised so the main idea here is that The thing with nerf is that it's very it's memory efficient Let's call it that way so the memory is cheap, but like the the inference is very expensive so if you want to run this thing if you want to get the Density and color from a certain point of the scene you're gonna have to do a forward pass which is way way Costlier compared to for example this voxel approach so we're here as you can see you have to explicitly keep the representation Like in a form of this of this of this cube, and that means it's memory expensive But like on the other hand like the actual like querying is very fast because you only have this shallow MLP part here so combining those two they get the representation here but this probably won't make any sense if you're not familiar with nerf or These other previous approaches, so I'm gonna do like a gonna do a quick digression here And explain how the nerf paper works in detail so nerf is very very interesting paper and Basically, there was a like a Cambrian explosion of papers that came after nerf that tried to to improve upon various various elements of nerf but here I'm going to explain the original architecture itself and The thing that's very funny and kind of there is a paradigm shift here and that is that Overfitting is actually good in nerf so nerf basically Overfits like heavily overfits to a 3d scene so you can see a bulldozer here as an example So it heavily overfits to that scene such that basically you can query this scene from a specific you take a specific point and you basically pick in like a like a like a ray like a direction vector and it will give you the pixel color from that position and that direction which is kind of Fascinating so you can do you can query this 3d scene after you've trained the nerf And I'm going to explain how that works in a second You can query it and get like these images you can see this one here or this one here You can get the same thing from from some in other spot like you can maybe place a camera here you place a camera here I'm gonna just Depict camera as this as this triangle and you get a novel novel image that you can just kind of Get by querying the MLP the multilayer perceptron that was trained and overfitted to this 3d scene So let me try and explain very briefly how how nerf works so you basically have a Sparse sampling of the 3d scene what I mean by that is that you have images such as this one So this is image number one taken from this particular location here. Let me try and change the color So we have from this location There was a camera here and somebody took the image and then there was a camera here and we have associated image So we have associated images and poses and it's very sparse like you maybe have 30 or 40 different images of this scene and Now what nerf does is the is the following you train the nerf the following way Basically nerf takes as an input this XYZ. So that's a coordinate So that's basically a position and so I want to so for example, I want to place a camera here So that's the position of the camera and then you have the theta and the Phi which are the angles They basically determine the the like a D array the vector in the space starting from that particular point in space and so once you input that that tuple that n-tuple into MLP It basically outputs like the three-dimensional vector that represents the the color so the RGB vector and this sigma thing which is called density of volume density basically so what volume density represents is basically the Probability that the ray has stopped at that particular point in in in the scene so think of that like this if the ray was if you had a ray like starting from here and Radiating through the scene and passing through the bulldozer Once the ray touches the surface of this of this part of the bulldozer and enters the actual bulldozer You'd expect the Sigma to go up because basically You will not be able to visualize that point because it's internal to the bulldozer and so the volume density kind of spikes up and you can see those Rays, so this is a visualization of the thing I was just mentioning so you take a ray like ray one So that may be this this red line here so that red line here and you can kind of visualize the the volume density here and now how nerf works is You take a bunch of points along a certain rate like you can see these these black dots here you query the MLP so this thing that's like a part of nerf and so that basically MLP is the thing that learns inside of this nerf framework and You query it alongside all of these points and you get the RGB Sigma Vectors and then what you do is using Sigma you do a weighted average And you get the final color and that's how they they kind of visualize that here so you basically will have some aggregate of these values and you're gonna form a color and Then you're gonna do a simple MSC loss so mean square error Using the ground truth pixel value so the you have the ground truth because you have the image So for example here, it's gonna be let me change the color again. Let me see like let me pick like yeah blue whatever so you'll have a like a yellow pixel here and Whatever came from this summation here you want to make sure to push it towards that point here and That's the whole trick so how you train like nerf is you take a bunch of these rays so have remember we have a sparse Sampling of this 3d scene. We'll have a bunch of these images We'll take randomly will take some rays emitting from from the cameras from from those images And we're gonna do we're gonna like do this sampling alongside those rays We're gonna form these colors as you can see here And we're gonna do the MSC loss and we're going to do this only like we're gonna basically overfit like crazy to this 3d scene So once that's done you can basically query nerf from some unseen position like as I said you can maybe place a camera here Or but where whatnot and you can get a 2d representation of this 3d scene from that particular pose And that's that's amazing. I mean you have this MLP this multilayer perception that Has encoded the whole 3d scene so every single information about that scene inside of the weights of that MLP So that's kind of crazy and an interesting paradigm if you've never seen nerf before So but in a nutshell that's it you have an MLP you take a bunch of rays You you query the the MLP alongside certain points You do the MSC loss, and that's how you train the nerf And it took I think the in the original paper somewhere around like seven eight hours Wall wall clock time to train this thing to you to kind of memorize a single scene. That's cool here you can see the the thing I was mentioning about how we actually accumulate the colors alongside along this particular Ray and the whole trick is to form these coefficients ti's change the color here So ti's are just this sum And we have the minus here and the exponent So it's exponential function and so what you can think of that you can think of this the following way So as I said once you get inside of the bulldozer The thing is this ti is gonna be very very small and that means that the color Inside of the bulldozer is gonna be weighted with a very small number here So that's gonna be approaching zero which means that the final color this C het Is gonna be heavily influenced by the first points that are close to the surface of the bulldozer Whereas the other points are gonna have less influence which makes a lot of sense So again this deltas are just the this distances let me zoom in here So Delta J's are just the distance between these successive points here that we are sampling and The sampling here seems like to be uniform, but in reality They are using stochastic sampling so that we can query the nerf the MLP later on Continuously across this whole space and you can see basically that as Sigma goes up. That means we are entering bulldozer This thing is gonna be bigger and bigger and because we have a minus sign. We just have an exponential When you have the minus That means it basically means we're gonna this thing is gonna converge to zero and that's what I just explained So hopefully that that makes a bit more sense how not nerf works So that's one approach and that's called implicit scene representation And it's implicit because the whole knowledge of the 3d scene is encoded inside of the weights of this Multi-layer perceptron network and that's very cool a bad thing about this approach Is that even though the whole scene is encoded in a relatively small number of weights if you want to render a single pixel So imagine we are now here and we don't have the actual pixel value. We'd have to Query the MLP across all of these points So that's a single forward pass times and because we have M point sampled along the ray In order to get a single pixel in the image space and we have to repeat that for every single pixel in the image space You can you can imagine that's kind of time intensive. Okay So that's where these second family of approaches come into play What they do instead is they explicitly keep the information about the 3d space as you can see here visualized in form of this cube and That means this approach is expensive memory wise, but it's much cheaper to do an inference Here in this model because you don't have to do a forward pass. You already have it explicitly store You can just kind of average it out and get the color of the pixel So that's everything I want you to take out from from from this image and explanation by the way I've put here inference prediction because there is a lot of Just to point out that people are very sloppy with terminology in general So we should be using probably prediction would be the correct terminology here not inference inferences Like it is more came from the context of probabilistic models And yeah, but people use these two interchangeably even though in reality We should be using the word prediction instead of inference, but yeah Super small digression there. Okay So now that we know the trade-offs, let's see what the actual paper brought to the table So we can see here what they did instead of storing every single Like RGB and the volume density in the scene They just have these three planes and even though these look like slices of paper They actually have the depth dimension because they'll have features here like maybe 32 features Alongside this dimension and every plane obviously will have like the depth. So this one we have here and we'll have here Maybe through the 32 features, but that's still way less compared to this Model here that has this so it's I think they call it. Yeah explicit voxel grid approach and Again, so it's memory It's more efficient memory wise and it's also more efficient time wise because as you can see here to find the RGB So to get to get the RGB value the color and the volume density We just have to pass it through this very shallow MLP and compare that to nerf where you have this huge huge Like MLP, I mean huge. I mean relatively big compared to this approach here So now the question remains so that's all nice and sweet but the question is is this as expressive as as as these models here and The answer will turn out to be yes, but before that I forgot to mention how this is constructed So if you want to query at the scene and get the the color at specific point in the scene What do you do you have this point you basically project it onto these planes. That means you have a In our case, we'll have a 32 dimensional vector here So it's going to be 32 dimensional vector and then extract the vector from here and from this projection So this is maybe XY plane and you just basically add them up and then you pass them through the a couple of fully connected Layers, basically a multi-layer perceptron here. So that's how the thing works now We need to make sure that this is as expressive as the two other approaches and that by creating this more efficient model We haven't sacrificed the quality and it turned out turns out that we have not as you can see here at the tree plane approach So that's the one here has a higher quality. The letters are much more like Readable compared to the two other approaches, which is very very cool That means we we've kept the expressive power of the two approaches and we made the model more efficient So what has happened here is that they are basically in this particular example The the trainable weights of this model are these weights Contained in the planes as well as the weights of this fully connected like of this fc layers and the the the way they they kind of Trained the tree plane model is the same way I just explained for nerf so they use the same approach to train and overfit on a particular scene in this case They have this scene called family scene so they train this this this component here the same way as you train nerf and they have to tweak these weights in the In these three planes and it's gonna change in the actual final model that I'm gonna explain right now By the way short aggression As I said these two papers would be useful to have some prior knowledge on and hopefully I gave you Enough context for you to understand the rest of this paper, but the third paper We're gonna find very useful is style again, so I guess the version one and two will suffice These kind of had incremental Improvements upon upon the first version of style again, which had amazing results So let's get to the actual final pipeline I've explained we now understand this part here We understand this whole part of the pipeline So let me kind of Encircle it in square it here. Whatever the name is so you can see we have these three planes we get the color and the density and Because we have the camera parameters 25 scalars We basically can Render the image from whatever pose in the scene so that part we understand and by the way It's 25 because it's 16 for the extrinsics matrix plus 9 for the intrinsics matrix extrinsics means that it basically just maps from the global from the world coordinate system into the camera coordinate system and Intrinsics are there to map from the camera coordinate system into the image space So there's some basic computer vision theory for you You don't have to worry too much about it But basically these 25 numbers tell you how the camera is placed and tell you certain properties About that camera so how it's placed in the scene and certain properties. So the approach will basically be to You're in the scene you're basically in a 3d scene you place a camera out somewhere And what you do is you emit bunch of rays and you sample bunch of points? Alongside those rays, so let me just kind of change the color So we'll sample bunch of points and all those points So when you query those you'll get as the output will get you get the color and the density the volume density And then you combine them in the the smart way I just showed you to get this final image here, which is 128 times 128 times 32, which is different So it's not an RGB image and the reason they've done it like this is because it's more Expressive and we'll see the details a bit later, but that's so that that's this component of the system now Let's switch our attention to the first part here What happens is they basically reuse the generator from the style again paper I mentioned so It's your classical again architecture all the upbeat very very smart one where as you can see here We have a latent of 512 scalars. So you have you have certain vector like some some noise vector Or I think it's actually trained in the case of style again You input that into the mapping vector the mapping vector creates these intermediate representations which we can then use to condition the Generator here which as you can see is up sampling in every single layer. We're doing some like we have some Deconvolution so-called deconvolution layers and we end up with this image 256 by 256 that's a spatial resolution and then we have 96 features And what we do is we just take the 96 features and we split it into three planes that each have a 32 Features per plane and we kind of just visually represent those here and then we know this part here now Now the interesting party I want you to see here is that camera prams are used to condition the mapping network Because that helps the generator decouple certain post correlated features and In general increase the expressivity of the of the generator I'm gonna probably show you a bit later what I mean by the correlation between pose and and and features But basically what it means, let me see whether I can find the the maybe somewhere in the appendix Basically people tend to smile when they're facing the camera up front and they Tend not to smile when they're not facing the camera. Let me see whether I can find that curve. Oh, yeah Okay, I found it here. So basically you can see here the plot that tells you so y'all is just basically tells you Tells you how rotated the head is alongside this let's call it XY plane and you can see that when you're facing the camera Like up front the the percentage of smiles is way higher when compared to you when when the head is turned Away from the camera and the reason is fairly simple when people know that they are being photographed They smile and when they are unaware they just being themselves and they're not smiling because Let's face it. This world is not that happy right? It's very it's a very sad world Anyways Enough with very very bad jokes. Let me go back to where I stopped So we are here trying to understand this whole pipeline Hopefully now I managed to to kind of break down this this first component of the pipeline Which is based off of style again The second part is based off of I said some combination of nerf and those explicit grid methods And finally, we have the third component here and let me now try and address This part and then you'll understand the complete method So what I've done here is Once they do the neural rendering and they have this 128 128 32 volume If they extract the first three channels Those will actually be representing the the actual image So those will contain by the way of training those will contain the down sampled version of the final output image And so what they do now is they up sample the image. So they'll up sample the first three channels using basically Bilinger so bilinear let me try and write it down bilinear transformation, which is not a learnable transformation That's just your your classical Digital image processing up sampling method nothing fancy there Secondly, they'll take this whole volume the 32 all of the 32 channels and they're going to pass it into this super resolution module Which is going to up sample the image and uh outcomes the 512 by 100 by 512 RGB image so three channels and again, we have some conditioning here. They're using these intermediate Latents to condition the the super resolution module Not completely sure whether that thing Whether they did some up relations, but like it makes sense that the information used to generate the original image I mean, it's not the original image. It's just these features that are then neural render. So yeah, i'm not sure whether this was Ablated but like whether it will work as good without this part, but yeah, it works like this And what happens now is they just take and concatenate the these the up sampled image and this super Like this image that was passed through the super resolution module And then they pass it into this style again Two version of discriminator, which is your your your classic GAN loss you're trying to figure out whether the image is real or not real Because you have some data set you basically have some data set of real images that's going to help you figure out This fact whether it's real or not Additionally, we are again conditioning the the discriminator using the camera parameters so the extrinsics and the intrinsics Now this is called what they call this is the Double discriminator or something like that. We're going to see the terminology a bit later, but the point is They are going to by concatenating and having six channels here. What they'll have to do is the following So they'll take some some real image. So this is a real image and then they're going to down sample that image like this and now they're going to use and That basically means that this discriminator is going to be able to use this image That basically means that this discriminator is going to force this small this new this this neural rendered image To be the same to be realistic as these down sampled real images and it's going to force Like the big image to be realistic as the big image is here And all of these constraints help the model learn how to to be consistent In in various poses and one of the crucial parts is actually conditioning on these camera parameters I forgot to mention how they they actually have so some data sets do have this data So the extrinsics and intrinsics In other data sets, basically they make the made an assumption of how intrinsics are What intrinsics make sense? We're going to see that in the appendix a bit later And they use some off-the-shelf detectors to figure out the pose So that that means to figure out the extrinsic component and that's how basically they train this whole thing So yeah, so we they don't have the 3d scans, but they do have the camera parameters. Keep that in mind. Okay Okay now That should be pretty much it. You know, understand should understand how the whole pipeline work But let me go through some additional details So here at this table, they just show the thing I mentioned about the trade-off between speed and memory That this novel approach is both faster and more memory efficient One thing worth worth mentioning is that the three plane representation is capable of representing this complex scene I'll be without view dependent effects That's what I mean by that is that in nerf you can change those theta and phi angles and get different color depending on how You're you're viewing the scene and that kind of helps you model the glossy effects the non-laboration I think that's the fancy way from the computer graphics world um so basically Let me let me see whether yeah, there is a Visualization of nerf somewhere here. So here it is. You can see that the color here Depends on the position and on the direction as well So the direction meaning the theta and phi angles so even if you're So even if you're in the scene in a certain position like here depending on how you kind of Direct the ray from that point you're going to have different colors Because of the way that nerf is set up here Whereas as you can see here the density the volume density is not a function of the direction Which makes sense because density just tells you whether the ray will stop at that particular point in the in the scene And that should not depend on the direction So if you're inside of bulldozer you're inside and it doesn't matter the direction This density should be high because you're inside of an object. That's not unless it's transparent Okay But yeah modeling transparent objects. That's a I guess a separate league Let's continue so I explained that part now I mentioned the the consistency that the double discriminator brings to the table now. Let's let's see what they set here First thing they say here is that the real images are fed into the discriminator are also processed by concatenating each of them With an appropriately blurred copy of itself So what I mean by that is that once you once you have this small image Once you down sampled it you'll have to up sample it again And by doing that you're gonna lose some details and the output image is gonna be somewhat blurred In order to be able to concatenate it and create this six channeled Like input volume so so this is the real image It has three channels and have certain spatial dimension and this this is the up sampled So we first down sample and then up sample the image In order to mimic the process here, okay So what I say here is that they are just kind of blurring and Basically, it should not be So the way they are probably approaching this is If you do like take an image of the full size and then you don't sample it and then you up sample it again I'm fairly sure that if you go into the spectral domain You can you can just devise a kernel a certain kernel like a kernel like this Like a kernel like in the sense in convolutional neural networks You can devise a kernel that's gonna modify the specter spectrum of this of this image in such a way that we get a blurred We get the same effect as if we were to down sample explicitly and up sample again But that's just a implementation detail that would probably make this process a bit more like efficient and optimized Okay, so the terminology I was I was trying to to get at was the dual dual discrimination That's how they call it and it not only encourages the final output to match the distribution of real images But also offers additional effects. So it encourages the neural rendering to match the distribution of thousand sampled real images So that's remember why so neural rendering to match the distribution of down sampled real images So that's because as you can see here, so this is the neural rendered image. So this thing here So as you recall we basically took the first three channels and we get this and now once we up sample this We are basically making sure that this thing looks as close as this real real image that was down sampled and up sampled Okay, so that's why we we have this consistency constraint um, and It encourages the super resolved images to be consistent with the neural rendering And what they mean by that is that basically this image here is going to be encouraged to be looked the same as this image here Which means that the super resolution is going to do its job Basically, it's going to just up sample the image without modifying the pose or the appearance or whatnot of this of this of this output image. Okay So that's it. Let's now continue and let's see some results. Results are fairly stunning You can see the identity of this of this uh, a man looking uh frontally And then when we change the pose the you can still say you can still tell that this is the same person and similarly here So this thing with the eyes is a bit creepy. It doesn't look like this person should be watching at the camera Looking at a camera. It looks like the eyes should be going there or something like that. But yeah Looks kind of weird. We're gonna see why that is a bit later Um, basically there are some problems where where the underlying geometry of the eye socket is concave instead of convex Which is creepy. Um They compare with the baseline models and as you can see every one of these has some problems either the geometry is super Bad or it's super super like very like smooth Um, and here it's also you can see it's the the image does not look realistic the the tooth The teeth here are are smoothened out and the geometry as well. And here is the the this method the proposed method looks way better Compared to the baselines just qualitatively analyzing these images Um Quantitative analysis show the same thing comparing with these baselines looking at a bunch of different metrics I'm not gonna get into what these metrics are fid basically a fresh raise inception score Um, uh, like depth and they're just comparing geometry and the quality of the uh, generating images and you can see that the uh, the the This approach is way better compared to the baselines. They also compare the speed the number of frames per second and you can see that It's lagging a little bit behind g-roff and some of those methods, but as you saw the quality is way better So that's kind of decent trade-off finally. They they use this fax smile. Um, Like sdd, I guess it's a standard deviation. I think this just kind of shows that the the Multi-view consistency is better when you use the dual discrimination and the Adding the generator pose conditioning the gpc to the model, so that's the basically The the camera parameters that we were feeding into generator to condition it. So that was this part here So using this conditioning on the parameters the generator the mapping network part of the generator That helps a lot incur this this This consistency, okay Let's continue and See some results. So this is i'm going to skip this this is from the style again. This is very Already very very clear from the style again papers You can mix up the style from the row. So this person here dictates the style and these persons in the column Dictate the general geometry and appearance you can see they're kind of mixing up this this kid here You can see how the style of the kid the style of the kid is changing depending on the leftmost column. Okay I thought explaining how this Pti works in more details, but it's gonna take it's gonna make it a little bit more complicated Details, but it's gonna take it's gonna make this video too long So I can maybe cover this separately in one of the next videos But basically what I managed to do is given the image The desired target image they managed to find the latent That generates that that image and then they can basically modify and edit the image semantically Inside of that latent space. So that's very very cool So what you can notice here I mentioned the concaveness of the eyes and that's the reason why it always looks like the eyes are looking directly into us into yeah Directly into our eyes, whatever is because you can see that the eye sockets are kind of concave here I think they they mentioned that somewhere down here. Let me let me find it So furthermore ambiguities that can be explained by geometry remain unresolved For example by creating concave eye sockets the generator creates the illusion of eyes that follow the camera an incorrect interpretation Though the renderings are view consistent and reflect the underlying geometry. Okay I'm not gonna dig deeper into that but um, i'm just gonna First address this paragraph here because I think it's also important not that like related to this concrete paper But I think it's a in general. I'm just yeah, I have this are these paragraphs useful Uh asking myself whether this is useful i'm gonna read it through Read it out loud so you can you can judge yourself So the single view 3d reconstruction or style mixing applications could be misused So it could be misused for generating edited imagery of real people. Okay, we are all familiar with deep fakes with what the GANs are in general kind of Like a problematic in that sense such misuse of image synthesis techniques poses a societal threat And we do not condone using our work with the intent of spreading misinformation or tarnishing reputation. I mean That's also kind of should be obvious authors do not want their paper their work to be used maliciously I don't see any added information in these sentences And finally, we also recognize a potential lack of diversity in our faces results stemming from implicit biases of the data sets We process and this like this same sentence could be copy pasted into so many different papers like Let's take let's replace this 3d GAN paper. Let's replace it with uh, like a gpt3 Or whatever your favorite language model is now in 2021 maybe without 2.0 or something. I don't know And basically so it's also can be misused and the author and secondly the authors condone using their work maliciously so that's also true and it also is true that depending on the biases in the they basically use the the the Text data set that was collected from the internet and reddit which has a lot of biases for like various various things like uh For example muslims being associated with crim like with terrorism stuff like that So some bad societal biases. So the point is Whether we should just be copy pasting this everywhere because this can be applied for every single paper Yeah, I don't mean anything bad by by saying all of this I just want to kind of raise the concern of whether this is going to be of any practical use paragraphs such as this one I'm not sure. I'm not smart here. I'm just I'm just uh, yeah speaking out loud here Um, finally, let me just find one additional thing here in the appendix Um, it has to do with the intrinsics Okay, here it is They say that we assume Fixed so the for the ffhq data set which is the data set of those high resolution human faces Not the cats the cats are the this is the cat's data set So they say the via zoom fixed camera intrinsics across the entire data set with a focal length of four point twenty six Times the image width equivalent to standard portrait lens. So that means that that's kind of assumed So that means it might not be correct. I'm not sure how the data set was collected So it may happen that different images were collected with different cameras which means this would not hold as a as a hypothesis and And they are using certain off-the-shelf detectors to figure out the extrinsics So I haven't seen not sure I may be be making a mistake here, but I haven't seen any Ablation on how on how robust the pipeline is to the errors in the this and this If this turns out not to be the case that this is a correct intrinsics for this data set Yeah, I just thought kind of mentioning that Also, one thing I'm not completely sure is how their Generating these geometry visualizations. I can assume that let me just go back to the pipeline here It's going to be easier to explain what I mean Okay, it's a long long paper, okay, what is probably happening is that they are using the density information So because so they pass array through the scene they have the density Information and they can kind of do a threshold and aggregate that density information So maybe these charts will be even better So they have something like this and then they can get just kind of threshold the the density and then use that to generate the Geometry, but I'm not sure whether I'm fairly sure that we have a lot of noise and glitching So it'd be nice to kind of Yeah for people who do not do this every day including me It'd be nice to have some hint of how stuff is done because yeah Understanding how geometry is actually visualized seems to be fairly important Anyways, if somebody knows exactly how this is done, feel free to comment down below and hopefully you like this video Let me know whether the lack of camera is Ruining the experience for you guys. So i'll make sure to in any case bring it back Very soon and create my youtube studio here in london Hopefully in like a couple of weeks in any case Subscribe if you if you like this video share out the the video with friends and join the discord community Because there is a lot of awesome cool people there which are willing to help each other out. So So consider joining the the community there and until next time. Bye. Bye
[{"start": 0.0, "end": 6.74, "text": " What's cracking guys? In this video I'm going to cover a very interesting paper and as a quick note"}, {"start": 6.74, "end": 11.96, "text": " I still don't have as you can see I don't have the video and the YouTube studio set up yet"}, {"start": 11.96, "end": 17.96, "text": " So I'm gonna have to only show you the paper for the time being hopefully I'm going to solve that very quickly"}, {"start": 18.48, "end": 25.0, "text": " But basically I'm covering this novel very interesting paper called efficient geometry aware"}, {"start": 25.0, "end": 29.7, "text": " 3d generative adversarial networks or GANs for short"}, {"start": 30.5, "end": 34.22, "text": " from this beautiful group of people from Stanford and"}, {"start": 34.82, "end": 41.879999999999995, "text": " Nvidia as a quick knit I think that like the very fact that this is 3d"}, {"start": 41.879999999999995, "end": 44.260000000000005, "text": " I mean, I think this is a pleonasm because"}, {"start": 45.1, "end": 49.5, "text": " 3d should imply the geometry aware part of this title"}, {"start": 49.5, "end": 57.980000000000004, "text": " But yeah in any case I guess it makes makes a more interesting title okay before I start digging into the actual paper"}, {"start": 58.019999999999996, "end": 63.9, "text": " Let me show you their video. We present a new 3d GAN that produces photorealistic images"}, {"start": 64.34, "end": 66.7, "text": " that are both multi-view consistent and"}, {"start": 69.78, "end": 71.78, "text": " Geometry aware"}, {"start": 73.3, "end": 79.02, "text": " Our approach enables unsupervised learning of high fidelity 3d representations without any"}, {"start": 79.02, "end": 81.02, "text": " explicit 3d or"}, {"start": 81.02, "end": 83.02, "text": " multi-view supervision"}, {"start": 86.74, "end": 91.97999999999999, "text": " So you can see results are fairly impressive because they are they are multi-view consistent"}, {"start": 91.97999999999999, "end": 95.89999999999999, "text": " You can see that even when you're changing the poses the appearance and the identity"}, {"start": 96.97999999999999, "end": 100.28, "text": " Remain the same which is kind of tricky for GANs to achieve"}, {"start": 100.74, "end": 106.02, "text": " And you can see that so here what's interesting is you can see let me stop it here"}, {"start": 106.02, "end": 109.38, "text": " you saw that the texture is kind of sticking and"}, {"start": 110.1, "end": 115.74, "text": " So when the cat is moving when the pose is changing you can see that the fur is kind of flickering or something"}, {"start": 115.74, "end": 121.02, "text": " So that stickiness problem. I think it's called like that like texture sticking problem was solved in one of the"}, {"start": 122.34, "end": 125.17999999999999, "text": " Newer style GAN papers if I'm not wrong"}, {"start": 125.82, "end": 131.3, "text": " Okay, in any case let's get back to the paper. We saw the the visualizations there and"}, {"start": 131.3, "end": 136.22, "text": " Now let me try and explain why this paper is special and very interesting so"}, {"start": 136.78, "end": 143.62, "text": " First thing they say here is that they train this whole thing this GAN without any ground truth 3d scans or"}, {"start": 144.10000000000002, "end": 152.02, "text": " Multi-view supervision so that means they just have 2d images basically and they manage to achieve this this"}, {"start": 152.78, "end": 157.18, "text": " Like basically multi-view consistency as you can see here the pose here changes"}, {"start": 157.18, "end": 162.82, "text": " But the identity remains the same and they also managed to recover the underlying 3d geometry"}, {"start": 163.38, "end": 169.32, "text": " Of the scene which is very cool now in order to understand how this"}, {"start": 169.86, "end": 173.74, "text": " Like paper works. We have to start with this novel"}, {"start": 174.66, "end": 181.76000000000002, "text": " Explicit implicit representation they devised called a tree plane. I think it's called tree plane representation or something"}, {"start": 182.18, "end": 183.58, "text": " but basically"}, {"start": 183.58, "end": 185.58, "text": " it's a combination it's a"}, {"start": 185.58, "end": 189.94000000000003, "text": " Somewhere in the middle between nerf, and I'm gonna briefly"}, {"start": 189.94000000000003, "end": 196.56, "text": " I'm gonna shortly explain what nerf exactly is and these like explicit voxel approaches"}, {"start": 197.46, "end": 203.86, "text": " And when you combine the good sides of this approach and the nerf approach you get this novel representation"}, {"start": 204.14000000000001, "end": 206.14000000000001, "text": " That they've devised"}, {"start": 206.22000000000003, "end": 208.82000000000002, "text": " so the main idea here is that"}, {"start": 209.54000000000002, "end": 213.32000000000002, "text": " The thing with nerf is that it's very it's memory efficient"}, {"start": 213.32, "end": 219.12, "text": " Let's call it that way so the memory is cheap, but like the the inference is very expensive"}, {"start": 219.12, "end": 222.51999999999998, "text": " so if you want to run this thing if you want to get the"}, {"start": 223.48, "end": 229.68, "text": " Density and color from a certain point of the scene you're gonna have to do a forward pass which is way way"}, {"start": 230.12, "end": 237.6, "text": " Costlier compared to for example this voxel approach so we're here as you can see you have to explicitly keep the representation"}, {"start": 237.6, "end": 243.68, "text": " Like in a form of this of this of this cube, and that means it's memory expensive"}, {"start": 243.76, "end": 250.88, "text": " But like on the other hand like the actual like querying is very fast because you only have this shallow"}, {"start": 251.04, "end": 255.64, "text": " MLP part here so combining those two they get the representation here"}, {"start": 255.84, "end": 260.64, "text": " but this probably won't make any sense if you're not familiar with nerf or"}, {"start": 261.36, "end": 266.21999999999997, "text": " These other previous approaches, so I'm gonna do like a gonna do a quick digression here"}, {"start": 266.22, "end": 271.1, "text": " And explain how the nerf paper works in detail"}, {"start": 271.86, "end": 273.08000000000004, "text": " so"}, {"start": 273.08000000000004, "end": 275.34000000000003, "text": " nerf is very very interesting paper and"}, {"start": 276.54, "end": 284.06, "text": " Basically, there was a like a Cambrian explosion of papers that came after nerf that tried to to improve upon various various elements of nerf"}, {"start": 284.38000000000005, "end": 287.78000000000003, "text": " but here I'm going to explain the original architecture itself and"}, {"start": 288.46000000000004, "end": 292.62, "text": " The thing that's very funny and kind of there is a paradigm shift here"}, {"start": 293.62, "end": 295.62, "text": " and that is that"}, {"start": 295.62, "end": 298.18, "text": " Overfitting is actually good in nerf so nerf"}, {"start": 298.9, "end": 300.1, "text": " basically"}, {"start": 300.1, "end": 305.98, "text": " Overfits like heavily overfits to a 3d scene so you can see a bulldozer here as an example"}, {"start": 306.42, "end": 309.9, "text": " So it heavily overfits to that scene such that"}, {"start": 310.5, "end": 314.78000000000003, "text": " basically you can query this scene from a specific you take a specific point and you"}, {"start": 315.78000000000003, "end": 321.82, "text": " basically pick in like a like a like a ray like a direction vector and it will give you the"}, {"start": 321.82, "end": 327.74, "text": " pixel color from that position and that direction which is kind of"}, {"start": 327.98, "end": 332.02, "text": " Fascinating so you can do you can query this 3d scene after you've trained the nerf"}, {"start": 332.02, "end": 334.02, "text": " And I'm going to explain how that works in a second"}, {"start": 334.14, "end": 340.06, "text": " You can query it and get like these images you can see this one here or this one here"}, {"start": 340.06, "end": 346.0, "text": " You can get the same thing from from some in other spot like you can maybe place a camera here you place a camera here"}, {"start": 346.0, "end": 346.94, "text": " I'm gonna just"}, {"start": 346.94, "end": 352.66, "text": " Depict camera as this as this triangle and you get a novel novel image that you can just kind of"}, {"start": 353.5, "end": 360.38, "text": " Get by querying the MLP the multilayer perceptron that was trained and overfitted to this 3d scene"}, {"start": 360.78, "end": 367.62, "text": " So let me try and explain very briefly how how nerf works so you basically have a"}, {"start": 368.02, "end": 373.38, "text": " Sparse sampling of the 3d scene what I mean by that is that you have images such as this one"}, {"start": 373.38, "end": 378.9, "text": " So this is image number one taken from this particular location here. Let me try and change the color"}, {"start": 378.98, "end": 380.7, "text": " So we have from this location"}, {"start": 380.7, "end": 386.4, "text": " There was a camera here and somebody took the image and then there was a camera here and we have associated image"}, {"start": 386.4, "end": 392.26, "text": " So we have associated images and poses and it's very sparse like you maybe have 30 or 40"}, {"start": 392.82, "end": 394.94, "text": " different images of this scene and"}, {"start": 395.54, "end": 401.42, "text": " Now what nerf does is the is the following you train the nerf the following way"}, {"start": 401.42, "end": 406.94, "text": " Basically nerf takes as an input this XYZ. So that's a coordinate"}, {"start": 406.94, "end": 411.26, "text": " So that's basically a position and so I want to so for example, I want to place a camera here"}, {"start": 411.26, "end": 416.14000000000004, "text": " So that's the position of the camera and then you have the theta and the Phi which are the angles"}, {"start": 416.14000000000004, "end": 421.18, "text": " They basically determine the the like a D array the vector in the space"}, {"start": 421.94, "end": 430.54, "text": " starting from that particular point in space and so once you input that that tuple that n-tuple into"}, {"start": 430.54, "end": 432.42, "text": " MLP"}, {"start": 432.42, "end": 441.06, "text": " It basically outputs like the three-dimensional vector that represents the the color so the RGB vector and this sigma thing"}, {"start": 441.06, "end": 443.06, "text": " which is called"}, {"start": 443.06, "end": 444.38, "text": " density of"}, {"start": 444.38, "end": 449.3, "text": " volume density basically so what volume density represents is basically the"}, {"start": 449.78000000000003, "end": 457.26, "text": " Probability that the ray has stopped at that particular point in in in the scene"}, {"start": 457.26, "end": 462.86, "text": " so think of that like this if the ray was if you had a ray like starting from here and"}, {"start": 464.02, "end": 466.7, "text": " Radiating through the scene and passing through the bulldozer"}, {"start": 467.58, "end": 473.9, "text": " Once the ray touches the surface of this of this part of the bulldozer and enters the actual bulldozer"}, {"start": 474.09999999999997, "end": 478.02, "text": " You'd expect the Sigma to go up because basically"}, {"start": 478.98, "end": 484.65999999999997, "text": " You will not be able to visualize that point because it's internal to the bulldozer and so the volume density"}, {"start": 484.66, "end": 488.86, "text": " kind of spikes up and you can see those"}, {"start": 489.3, "end": 493.8, "text": " Rays, so this is a visualization of the thing I was just mentioning so you take a ray like ray one"}, {"start": 493.84000000000003, "end": 496.62, "text": " So that may be this this red line here"}, {"start": 497.18, "end": 501.34000000000003, "text": " so that red line here and you can kind of visualize the the"}, {"start": 502.1, "end": 506.06, "text": " volume density here and now how nerf works is"}, {"start": 506.62, "end": 512.0600000000001, "text": " You take a bunch of points along a certain rate like you can see these these black dots here"}, {"start": 512.06, "end": 516.2199999999999, "text": " you query the MLP so this thing that's"}, {"start": 516.9399999999999, "end": 518.9399999999999, "text": " like a part of nerf and"}, {"start": 518.9799999999999, "end": 524.0999999999999, "text": " so that basically MLP is the thing that learns inside of this nerf framework and"}, {"start": 524.42, "end": 529.9399999999999, "text": " You query it alongside all of these points and you get the RGB Sigma"}, {"start": 530.66, "end": 536.14, "text": " Vectors and then what you do is using Sigma you do a weighted average"}, {"start": 536.14, "end": 540.9399999999999, "text": " And you get the final color and that's how they they kind of visualize that here"}, {"start": 540.94, "end": 546.9000000000001, "text": " so you basically will have some aggregate of these values and you're gonna form a color and"}, {"start": 547.1, "end": 551.0600000000001, "text": " Then you're gonna do a simple MSC loss so mean square error"}, {"start": 552.22, "end": 557.58, "text": " Using the ground truth pixel value so the you have the ground truth because you have the image"}, {"start": 557.58, "end": 563.98, "text": " So for example here, it's gonna be let me change the color again. Let me see like let me pick like yeah blue"}, {"start": 563.98, "end": 567.74, "text": " whatever so you'll have a like a yellow pixel here and"}, {"start": 567.74, "end": 575.7, "text": " Whatever came from this summation here you want to make sure to push it towards that point here and"}, {"start": 576.46, "end": 583.22, "text": " That's the whole trick so how you train like nerf is you take a bunch of these rays so have remember we have a sparse"}, {"start": 583.5, "end": 585.98, "text": " Sampling of this 3d scene. We'll have a bunch of these images"}, {"start": 586.5, "end": 591.82, "text": " We'll take randomly will take some rays emitting from from the cameras from from those images"}, {"start": 591.82, "end": 597.02, "text": " And we're gonna do we're gonna like do this sampling alongside those rays"}, {"start": 597.02, "end": 599.8199999999999, "text": " We're gonna form these colors as you can see here"}, {"start": 599.8199999999999, "end": 607.54, "text": " And we're gonna do the MSC loss and we're going to do this only like we're gonna basically overfit like crazy to this 3d scene"}, {"start": 608.1, "end": 614.38, "text": " So once that's done you can basically query nerf from some unseen position like as I said you can maybe place a camera here"}, {"start": 614.38, "end": 621.28, "text": " Or but where whatnot and you can get a 2d representation of this 3d scene from that particular pose"}, {"start": 621.28, "end": 626.22, "text": " And that's that's amazing. I mean you have this MLP this multilayer perception"}, {"start": 626.22, "end": 627.5400000000001, "text": " that"}, {"start": 627.5400000000001, "end": 635.24, "text": " Has encoded the whole 3d scene so every single information about that scene inside of the weights of that MLP"}, {"start": 635.24, "end": 639.14, "text": " So that's kind of crazy and an interesting paradigm if you've never seen nerf before"}, {"start": 639.86, "end": 643.72, "text": " So but in a nutshell that's it you have an MLP you take a bunch of rays"}, {"start": 644.3000000000001, "end": 649.22, "text": " You you query the the MLP alongside certain points"}, {"start": 649.6600000000001, "end": 652.78, "text": " You do the MSC loss, and that's how you train the nerf"}, {"start": 652.78, "end": 656.9399999999999, "text": " And it took I think the in the original paper somewhere around like seven eight hours"}, {"start": 657.4599999999999, "end": 663.18, "text": " Wall wall clock time to train this thing to you to kind of memorize a single scene. That's cool"}, {"start": 664.1, "end": 668.6999999999999, "text": " here you can see the the thing I was mentioning about how we actually accumulate the"}, {"start": 669.3, "end": 671.9, "text": " colors alongside along this particular"}, {"start": 672.54, "end": 678.9399999999999, "text": " Ray and the whole trick is to form these coefficients ti's change the color here"}, {"start": 678.9399999999999, "end": 681.62, "text": " So ti's are just this sum"}, {"start": 681.62, "end": 684.34, "text": " And we have the minus here and the exponent"}, {"start": 684.86, "end": 690.66, "text": " So it's exponential function and so what you can think of that you can think of this the following way"}, {"start": 690.66, "end": 693.54, "text": " So as I said once you get inside of the bulldozer"}, {"start": 694.42, "end": 699.86, "text": " The thing is this ti is gonna be very very small and that means that the color"}, {"start": 700.34, "end": 705.02, "text": " Inside of the bulldozer is gonna be weighted with a very small number here"}, {"start": 705.02, "end": 709.92, "text": " So that's gonna be approaching zero which means that the final color this C het"}, {"start": 709.92, "end": 717.36, "text": " Is gonna be heavily influenced by the first points that are close to the surface of the bulldozer"}, {"start": 717.7199999999999, "end": 721.52, "text": " Whereas the other points are gonna have less influence which makes a lot of sense"}, {"start": 721.68, "end": 726.3199999999999, "text": " So again this deltas are just the this distances let me zoom in here"}, {"start": 726.4, "end": 729.7199999999999, "text": " So Delta J's are just the distance between these"}, {"start": 730.3199999999999, "end": 732.52, "text": " successive points here that we are sampling and"}, {"start": 732.8, "end": 736.8399999999999, "text": " The sampling here seems like to be uniform, but in reality"}, {"start": 736.84, "end": 741.9200000000001, "text": " They are using stochastic sampling so that we can query the nerf the MLP later on"}, {"start": 742.72, "end": 750.24, "text": " Continuously across this whole space and you can see basically that as Sigma goes up. That means we are entering bulldozer"}, {"start": 751.08, "end": 756.6, "text": " This thing is gonna be bigger and bigger and because we have a minus sign. We just have an exponential"}, {"start": 757.4, "end": 759.4, "text": " When you have the minus"}, {"start": 759.5600000000001, "end": 765.36, "text": " That means it basically means we're gonna this thing is gonna converge to zero and that's what I just explained"}, {"start": 765.36, "end": 768.86, "text": " So hopefully that that makes a bit more sense how not nerf works"}, {"start": 769.72, "end": 775.22, "text": " So that's one approach and that's called implicit scene representation"}, {"start": 775.22, "end": 780.94, "text": " And it's implicit because the whole knowledge of the 3d scene is encoded inside of the weights of this"}, {"start": 781.2, "end": 785.88, "text": " Multi-layer perceptron network and that's very cool a bad thing about this approach"}, {"start": 785.88, "end": 793.58, "text": " Is that even though the whole scene is encoded in a relatively small number of weights if you want to render a single pixel"}, {"start": 793.58, "end": 797.74, "text": " So imagine we are now here and we don't have the actual pixel value. We'd have to"}, {"start": 799.12, "end": 801.12, "text": " Query the"}, {"start": 801.12, "end": 802.88, "text": " MLP across all of these points"}, {"start": 802.88, "end": 807.12, "text": " So that's a single forward pass times and because we have M point sampled along the ray"}, {"start": 807.4000000000001, "end": 814.0600000000001, "text": " In order to get a single pixel in the image space and we have to repeat that for every single pixel in the image space"}, {"start": 814.0600000000001, "end": 817.0400000000001, "text": " You can you can imagine that's kind of time intensive. Okay"}, {"start": 818.2, "end": 821.76, "text": " So that's where these second family of approaches come into play"}, {"start": 821.76, "end": 829.28, "text": " What they do instead is they explicitly keep the information about the 3d space as you can see here"}, {"start": 829.28, "end": 831.28, "text": " visualized in form of this cube and"}, {"start": 832.2, "end": 839.24, "text": " That means this approach is expensive memory wise, but it's much cheaper to do an inference"}, {"start": 840.36, "end": 845.6, "text": " Here in this model because you don't have to do a forward pass. You already have it explicitly store"}, {"start": 845.6, "end": 848.7, "text": " You can just kind of average it out and get the color of the pixel"}, {"start": 848.7, "end": 853.1400000000001, "text": " So that's everything I want you to take out from from from this image and explanation"}, {"start": 853.38, "end": 854.0200000000001, "text": " by the way"}, {"start": 854.0200000000001, "end": 856.62, "text": " I've put here inference prediction because there is a lot of"}, {"start": 857.22, "end": 861.6800000000001, "text": " Just to point out that people are very sloppy with terminology in general"}, {"start": 861.6800000000001, "end": 866.58, "text": " So we should be using probably prediction would be the correct terminology here not inference inferences"}, {"start": 866.58, "end": 870.7, "text": " Like it is more came from the context of probabilistic models"}, {"start": 870.7, "end": 875.82, "text": " And yeah, but people use these two interchangeably even though in reality"}, {"start": 875.82, "end": 878.7800000000001, "text": " We should be using the word prediction instead of inference, but yeah"}, {"start": 879.58, "end": 881.58, "text": " Super small digression there. Okay"}, {"start": 882.0200000000001, "end": 888.0400000000001, "text": " So now that we know the trade-offs, let's see what the actual paper brought to the table"}, {"start": 888.1, "end": 893.2600000000001, "text": " So we can see here what they did instead of storing every single"}, {"start": 893.5, "end": 897.0200000000001, "text": " Like RGB and the volume density in the scene"}, {"start": 897.3000000000001, "end": 902.1, "text": " They just have these three planes and even though these look like slices of paper"}, {"start": 902.1, "end": 906.86, "text": " They actually have the depth dimension because they'll have features here like maybe 32 features"}, {"start": 907.94, "end": 913.9200000000001, "text": " Alongside this dimension and every plane obviously will have like the depth. So this one we have here and we'll have here"}, {"start": 914.74, "end": 919.6800000000001, "text": " Maybe through the 32 features, but that's still way less compared to this"}, {"start": 920.3000000000001, "end": 925.4200000000001, "text": " Model here that has this so it's I think they call it. Yeah explicit voxel grid approach"}, {"start": 926.1, "end": 927.34, "text": " and"}, {"start": 927.34, "end": 929.34, "text": " Again, so it's memory"}, {"start": 929.34, "end": 937.0600000000001, "text": " It's more efficient memory wise and it's also more efficient time wise because as you can see here to find the RGB"}, {"start": 937.0600000000001, "end": 941.5400000000001, "text": " So to get to get the RGB value the color and the volume density"}, {"start": 941.7800000000001, "end": 948.82, "text": " We just have to pass it through this very shallow MLP and compare that to nerf where you have this huge huge"}, {"start": 949.0600000000001, "end": 954.34, "text": " Like MLP, I mean huge. I mean relatively big compared to this approach here"}, {"start": 954.34, "end": 959.7, "text": " So now the question remains so that's all nice and sweet but the question is is this as"}, {"start": 960.0600000000001, "end": 962.86, "text": " expressive as as as these models here and"}, {"start": 963.34, "end": 967.9, "text": " The answer will turn out to be yes, but before that I forgot to mention how this is constructed"}, {"start": 967.94, "end": 973.9, "text": " So if you want to query at the scene and get the the color at specific point in the scene"}, {"start": 974.0600000000001, "end": 980.02, "text": " What do you do you have this point you basically project it onto these planes. That means you have a"}, {"start": 980.02, "end": 983.6999999999999, "text": " In our case, we'll have a 32 dimensional vector here"}, {"start": 983.6999999999999, "end": 989.14, "text": " So it's going to be 32 dimensional vector and then extract the vector from here and from this projection"}, {"start": 989.14, "end": 995.74, "text": " So this is maybe XY plane and you just basically add them up and then you pass them through the a couple of fully connected"}, {"start": 995.74, "end": 1000.42, "text": " Layers, basically a multi-layer perceptron here. So that's how the thing works now"}, {"start": 1001.02, "end": 1009.1, "text": " We need to make sure that this is as expressive as the two other approaches and that by creating this more efficient model"}, {"start": 1009.1, "end": 1016.9, "text": " We haven't sacrificed the quality and it turned out turns out that we have not as you can see here at the tree plane approach"}, {"start": 1016.9, "end": 1021.5, "text": " So that's the one here has a higher quality. The letters are much more"}, {"start": 1021.98, "end": 1023.02, "text": " like"}, {"start": 1023.02, "end": 1027.9, "text": " Readable compared to the two other approaches, which is very very cool"}, {"start": 1027.9, "end": 1033.46, "text": " That means we we've kept the expressive power of the two approaches and we made the model more efficient"}, {"start": 1033.46, "end": 1037.94, "text": " So what has happened here is that they are basically in this particular example"}, {"start": 1037.94, "end": 1042.66, "text": " The the trainable weights of this model are these weights"}, {"start": 1043.1000000000001, "end": 1047.54, "text": " Contained in the planes as well as the weights of this fully connected"}, {"start": 1048.18, "end": 1053.94, "text": " like of this fc layers and the the the way they they kind of"}, {"start": 1054.38, "end": 1058.6000000000001, "text": " Trained the tree plane model is the same way"}, {"start": 1058.6000000000001, "end": 1065.66, "text": " I just explained for nerf so they use the same approach to train and overfit on a particular scene in this case"}, {"start": 1065.66, "end": 1068.0600000000002, "text": " They have this scene called family scene"}, {"start": 1068.1000000000001, "end": 1073.78, "text": " so they train this this this component here the same way as you train nerf and"}, {"start": 1074.3000000000002, "end": 1077.3000000000002, "text": " they have to tweak these weights in the"}, {"start": 1078.22, "end": 1084.7, "text": " In these three planes and it's gonna change in the actual final model that I'm gonna explain right now"}, {"start": 1086.3000000000002, "end": 1088.3000000000002, "text": " By the way short aggression"}, {"start": 1088.3000000000002, "end": 1093.3400000000001, "text": " As I said these two papers would be useful to have some prior knowledge on and hopefully I gave you"}, {"start": 1093.34, "end": 1098.3, "text": " Enough context for you to understand the rest of this paper, but the third paper"}, {"start": 1098.3, "end": 1104.62, "text": " We're gonna find very useful is style again, so I guess the version one and two will suffice"}, {"start": 1105.22, "end": 1107.22, "text": " These kind of had incremental"}, {"start": 1108.02, "end": 1112.6599999999999, "text": " Improvements upon upon the first version of style again, which had amazing results"}, {"start": 1113.1799999999998, "end": 1115.62, "text": " So let's get to the actual final pipeline"}, {"start": 1116.62, "end": 1120.26, "text": " I've explained we now understand this part here"}, {"start": 1120.26, "end": 1123.3799999999999, "text": " We understand this whole part of the pipeline"}, {"start": 1124.14, "end": 1126.02, "text": " So let me kind of"}, {"start": 1126.02, "end": 1130.5, "text": " Encircle it in square it here. Whatever the name is"}, {"start": 1131.86, "end": 1135.1, "text": " so you can see we have these three planes we"}, {"start": 1136.06, "end": 1138.1, "text": " get the color and the density and"}, {"start": 1139.18, "end": 1142.36, "text": " Because we have the camera parameters 25 scalars"}, {"start": 1143.74, "end": 1145.74, "text": " We basically can"}, {"start": 1145.74, "end": 1151.7, "text": " Render the image from whatever pose in the scene so that part we understand and by the way"}, {"start": 1151.7, "end": 1155.52, "text": " It's 25 because it's 16 for the extrinsics matrix"}, {"start": 1156.34, "end": 1159.1, "text": " plus 9 for the intrinsics matrix"}, {"start": 1160.06, "end": 1167.14, "text": " extrinsics means that it basically just maps from the global from the world coordinate system into the camera coordinate system and"}, {"start": 1167.74, "end": 1173.04, "text": " Intrinsics are there to map from the camera coordinate system into the image space"}, {"start": 1173.04, "end": 1175.72, "text": " So there's some basic computer vision theory for you"}, {"start": 1175.72, "end": 1177.92, "text": " You don't have to worry too much about it"}, {"start": 1177.92, "end": 1184.04, "text": " But basically these 25 numbers tell you how the camera is placed and tell you certain properties"}, {"start": 1184.32, "end": 1190.32, "text": " About that camera so how it's placed in the scene and certain properties. So the approach will basically be to"}, {"start": 1191.84, "end": 1196.8799999999999, "text": " You're in the scene you're basically in a 3d scene you place a camera out somewhere"}, {"start": 1196.88, "end": 1202.8400000000001, "text": " And what you do is you emit bunch of rays and you sample bunch of points?"}, {"start": 1203.64, "end": 1206.16, "text": " Alongside those rays, so let me just kind of change the color"}, {"start": 1206.2800000000002, "end": 1209.92, "text": " So we'll sample bunch of points and all those points"}, {"start": 1210.24, "end": 1215.8000000000002, "text": " So when you query those you'll get as the output will get you get the color and the density the volume density"}, {"start": 1215.8000000000002, "end": 1218.6200000000001, "text": " And then you combine them in the the smart way"}, {"start": 1218.6200000000001, "end": 1226.0800000000002, "text": " I just showed you to get this final image here, which is 128 times 128 times 32, which is different"}, {"start": 1226.08, "end": 1230.4399999999998, "text": " So it's not an RGB image and the reason they've done it like this is because it's more"}, {"start": 1230.6399999999999, "end": 1236.6, "text": " Expressive and we'll see the details a bit later, but that's so that that's this component of the system now"}, {"start": 1236.76, "end": 1239.8799999999999, "text": " Let's switch our attention to the first part here"}, {"start": 1240.4399999999998, "end": 1246.8, "text": " What happens is they basically reuse the generator from the style again paper I mentioned so"}, {"start": 1247.48, "end": 1253.3799999999999, "text": " It's your classical again architecture all the upbeat very very smart one where as you can see here"}, {"start": 1253.38, "end": 1259.3400000000001, "text": " We have a latent of 512 scalars. So you have you have certain vector like some some noise vector"}, {"start": 1259.3400000000001, "end": 1263.0200000000002, "text": " Or I think it's actually trained in the case of style again"}, {"start": 1263.8400000000001, "end": 1268.9, "text": " You input that into the mapping vector the mapping vector creates these intermediate"}, {"start": 1269.3200000000002, "end": 1272.5, "text": " representations which we can then use to condition the"}, {"start": 1273.0, "end": 1279.6200000000001, "text": " Generator here which as you can see is up sampling in every single layer. We're doing some like we have some"}, {"start": 1279.62, "end": 1285.54, "text": " Deconvolution so-called deconvolution layers and we end up with this image"}, {"start": 1286.34, "end": 1291.2199999999998, "text": " 256 by 256 that's a spatial resolution and then we have 96 features"}, {"start": 1291.2199999999998, "end": 1298.4599999999998, "text": " And what we do is we just take the 96 features and we split it into three planes that each have a 32"}, {"start": 1298.8999999999999, "end": 1306.06, "text": " Features per plane and we kind of just visually represent those here and then we know this part here now"}, {"start": 1306.06, "end": 1313.94, "text": " Now the interesting party I want you to see here is that camera prams are used to condition the mapping network"}, {"start": 1315.34, "end": 1320.94, "text": " Because that helps the generator decouple certain post correlated features and"}, {"start": 1321.5, "end": 1324.78, "text": " In general increase the expressivity of the of the generator"}, {"start": 1325.34, "end": 1331.2, "text": " I'm gonna probably show you a bit later what I mean by the correlation between pose and and and features"}, {"start": 1331.2, "end": 1336.48, "text": " But basically what it means, let me see whether I can find the the maybe somewhere in the appendix"}, {"start": 1338.24, "end": 1343.68, "text": " Basically people tend to smile when they're facing the camera up front and they"}, {"start": 1344.04, "end": 1350.0, "text": " Tend not to smile when they're not facing the camera. Let me see whether I can find that curve. Oh, yeah"}, {"start": 1350.0, "end": 1357.38, "text": " Okay, I found it here. So basically you can see here the plot that tells you so y'all is just basically tells you"}, {"start": 1357.38, "end": 1366.5, "text": " Tells you how rotated the head is alongside this let's call it XY plane and you can see that when you're facing the camera"}, {"start": 1367.7, "end": 1375.38, "text": " Like up front the the percentage of smiles is way higher when compared to you when when the head is turned"}, {"start": 1375.7, "end": 1381.3000000000002, "text": " Away from the camera and the reason is fairly simple when people know that they are being photographed"}, {"start": 1381.6200000000001, "end": 1385.94, "text": " They smile and when they are unaware they just being themselves and they're not smiling because"}, {"start": 1385.94, "end": 1389.8600000000001, "text": " Let's face it. This world is not that happy right? It's very it's a very sad world"}, {"start": 1390.74, "end": 1392.18, "text": " Anyways"}, {"start": 1392.18, "end": 1396.3400000000001, "text": " Enough with very very bad jokes. Let me go back to where I stopped"}, {"start": 1397.14, "end": 1400.42, "text": " So we are here trying to understand this whole pipeline"}, {"start": 1401.54, "end": 1407.54, "text": " Hopefully now I managed to to kind of break down this this first component of the pipeline"}, {"start": 1408.26, "end": 1410.98, "text": " Which is based off of style again"}, {"start": 1410.98, "end": 1417.06, "text": " The second part is based off of I said some combination of nerf and those explicit grid methods"}, {"start": 1417.7, "end": 1422.18, "text": " And finally, we have the third component here and let me now try and address"}, {"start": 1422.74, "end": 1425.22, "text": " This part and then you'll understand the complete method"}, {"start": 1426.42, "end": 1428.42, "text": " So what I've done here is"}, {"start": 1428.82, "end": 1433.78, "text": " Once they do the neural rendering and they have this 128 128 32"}, {"start": 1434.58, "end": 1435.46, "text": " volume"}, {"start": 1435.46, "end": 1437.78, "text": " If they extract the first three channels"}, {"start": 1437.78, "end": 1441.46, "text": " Those will actually be representing the the actual image"}, {"start": 1441.46, "end": 1448.34, "text": " So those will contain by the way of training those will contain the down sampled version of the final output image"}, {"start": 1448.98, "end": 1456.02, "text": " And so what they do now is they up sample the image. So they'll up sample the first three channels using basically"}, {"start": 1456.8999999999999, "end": 1463.1399999999999, "text": " Bilinger so bilinear let me try and write it down bilinear transformation, which is not a learnable transformation"}, {"start": 1463.1399999999999, "end": 1464.98, "text": " That's just your your classical"}, {"start": 1464.98, "end": 1468.5, "text": " Digital image processing up sampling method nothing fancy there"}, {"start": 1469.54, "end": 1476.9, "text": " Secondly, they'll take this whole volume the 32 all of the 32 channels and they're going to pass it into this super resolution module"}, {"start": 1477.8600000000001, "end": 1484.58, "text": " Which is going to up sample the image and uh outcomes the 512 by 100 by 512"}, {"start": 1485.3, "end": 1491.14, "text": " RGB image so three channels and again, we have some conditioning here. They're using these intermediate"}, {"start": 1491.14, "end": 1494.9, "text": " Latents to condition the the super resolution module"}, {"start": 1495.6200000000001, "end": 1498.74, "text": " Not completely sure whether that thing"}, {"start": 1499.46, "end": 1506.8200000000002, "text": " Whether they did some up relations, but like it makes sense that the information used to generate the original image"}, {"start": 1507.46, "end": 1513.7800000000002, "text": " I mean, it's not the original image. It's just these features that are then neural render. So yeah, i'm not sure whether this"}, {"start": 1514.18, "end": 1515.46, "text": " was"}, {"start": 1515.46, "end": 1522.02, "text": " Ablated but like whether it will work as good without this part, but yeah, it works like this"}, {"start": 1522.42, "end": 1529.22, "text": " And what happens now is they just take and concatenate the these the up sampled image and this super"}, {"start": 1529.8600000000001, "end": 1534.02, "text": " Like this image that was passed through the super resolution module"}, {"start": 1535.14, "end": 1537.7, "text": " And then they pass it into this style again"}, {"start": 1538.5, "end": 1542.02, "text": " Two version of discriminator, which is your your your classic"}, {"start": 1542.02, "end": 1546.18, "text": " GAN loss you're trying to figure out whether the image is real or not real"}, {"start": 1546.82, "end": 1552.5, "text": " Because you have some data set you basically have some data set of real images that's going to help you figure out"}, {"start": 1553.06, "end": 1555.06, "text": " This fact whether it's real or not"}, {"start": 1555.94, "end": 1564.5, "text": " Additionally, we are again conditioning the the discriminator using the camera parameters so the extrinsics and the intrinsics"}, {"start": 1565.78, "end": 1568.26, "text": " Now this is called what they call this is the"}, {"start": 1568.26, "end": 1573.3, "text": " Double discriminator or something like that. We're going to see the terminology a bit later, but the point is"}, {"start": 1574.5, "end": 1579.46, "text": " They are going to by concatenating and having six channels here. What they'll have to do is the following"}, {"start": 1579.46, "end": 1584.66, "text": " So they'll take some some real image. So this is a real image and then they're going to"}, {"start": 1585.3, "end": 1587.3, "text": " down sample that image"}, {"start": 1587.3, "end": 1588.58, "text": " like this"}, {"start": 1588.58, "end": 1590.58, "text": " and now they're going to use and"}, {"start": 1591.14, "end": 1595.54, "text": " That basically means that this discriminator is going to be able to use this image"}, {"start": 1595.54, "end": 1602.1, "text": " That basically means that this discriminator is going to force this small this new this this neural rendered image"}, {"start": 1602.42, "end": 1608.26, "text": " To be the same to be realistic as these down sampled real images and it's going to force"}, {"start": 1609.3799999999999, "end": 1612.42, "text": " Like the big image to be realistic as the big image is here"}, {"start": 1613.46, "end": 1618.8999999999999, "text": " And all of these constraints help the model learn how to to be consistent"}, {"start": 1618.9, "end": 1625.3000000000002, "text": " In in various poses and one of the crucial parts is actually conditioning on these camera parameters"}, {"start": 1625.5400000000002, "end": 1629.94, "text": " I forgot to mention how they they actually have so some data sets do have this data"}, {"start": 1630.42, "end": 1632.5800000000002, "text": " So the extrinsics and intrinsics"}, {"start": 1633.38, "end": 1638.02, "text": " In other data sets, basically they make the made an assumption of how intrinsics are"}, {"start": 1639.0600000000002, "end": 1642.3400000000001, "text": " What intrinsics make sense? We're going to see that in the appendix a bit later"}, {"start": 1642.98, "end": 1646.42, "text": " And they use some off-the-shelf detectors to figure out the pose"}, {"start": 1646.42, "end": 1652.5800000000002, "text": " So that that means to figure out the extrinsic component and that's how basically they train this whole thing"}, {"start": 1652.74, "end": 1658.66, "text": " So yeah, so we they don't have the 3d scans, but they do have the camera parameters. Keep that in mind. Okay"}, {"start": 1660.8200000000002, "end": 1662.02, "text": " Okay now"}, {"start": 1662.02, "end": 1666.02, "text": " That should be pretty much it. You know, understand should understand how the whole pipeline work"}, {"start": 1666.02, "end": 1668.26, "text": " But let me go through some additional details"}, {"start": 1668.66, "end": 1673.78, "text": " So here at this table, they just show the thing I mentioned about the trade-off between speed and memory"}, {"start": 1673.78, "end": 1678.5, "text": " That this novel approach is both faster and more memory efficient"}, {"start": 1679.78, "end": 1686.26, "text": " One thing worth worth mentioning is that the three plane representation is capable of representing this complex scene"}, {"start": 1686.58, "end": 1689.3, "text": " I'll be without view dependent effects"}, {"start": 1690.1, "end": 1698.1, "text": " That's what I mean by that is that in nerf you can change those theta and phi angles and get different color depending on how"}, {"start": 1698.1, "end": 1704.1, "text": " You're you're viewing the scene and that kind of helps you model the glossy effects the non-laboration"}, {"start": 1704.1799999999998, "end": 1706.82, "text": " I think that's the fancy way from the computer graphics world"}, {"start": 1707.54, "end": 1708.74, "text": " um"}, {"start": 1708.74, "end": 1710.4199999999998, "text": " so basically"}, {"start": 1710.4199999999998, "end": 1712.74, "text": " Let me let me see whether yeah, there is a"}, {"start": 1713.76, "end": 1717.62, "text": " Visualization of nerf somewhere here. So here it is. You can see that"}, {"start": 1718.1799999999998, "end": 1720.1799999999998, "text": " the color here"}, {"start": 1720.34, "end": 1724.82, "text": " Depends on the position and on the direction as well"}, {"start": 1724.82, "end": 1729.3799999999999, "text": " So the direction meaning the theta and phi angles so even if you're"}, {"start": 1729.9399999999998, "end": 1734.02, "text": " So even if you're in the scene in a certain position like here"}, {"start": 1734.56, "end": 1736.6599999999999, "text": " depending on how you kind of"}, {"start": 1738.1799999999998, "end": 1741.78, "text": " Direct the ray from that point you're going to have different colors"}, {"start": 1742.74, "end": 1745.3, "text": " Because of the way that nerf is set up here"}, {"start": 1745.62, "end": 1750.74, "text": " Whereas as you can see here the density the volume density is not a function of the direction"}, {"start": 1750.74, "end": 1758.1, "text": " Which makes sense because density just tells you whether the ray will stop at that particular point in the in the scene"}, {"start": 1758.5, "end": 1760.82, "text": " And that should not depend on the direction"}, {"start": 1761.7, "end": 1765.94, "text": " So if you're inside of bulldozer you're inside and it doesn't matter the direction"}, {"start": 1766.74, "end": 1771.94, "text": " This density should be high because you're inside of an object. That's not unless it's transparent"}, {"start": 1772.74, "end": 1773.86, "text": " Okay"}, {"start": 1773.86, "end": 1776.82, "text": " But yeah modeling transparent objects. That's a I guess a separate league"}, {"start": 1776.82, "end": 1780.1799999999998, "text": " Let's continue so I explained that part now"}, {"start": 1781.22, "end": 1788.1799999999998, "text": " I mentioned the the consistency that the double discriminator brings to the table now. Let's let's see what they set here"}, {"start": 1788.6599999999999, "end": 1795.7, "text": " First thing they say here is that the real images are fed into the discriminator are also processed by concatenating each of them"}, {"start": 1795.86, "end": 1799.06, "text": " With an appropriately blurred copy of itself"}, {"start": 1799.46, "end": 1804.1799999999998, "text": " So what I mean by that is that once you once you have this small image"}, {"start": 1804.18, "end": 1807.22, "text": " Once you down sampled it you'll have to up sample it again"}, {"start": 1808.02, "end": 1813.0600000000002, "text": " And by doing that you're gonna lose some details and the output image is gonna be somewhat blurred"}, {"start": 1813.54, "end": 1817.78, "text": " In order to be able to concatenate it and create this six channeled"}, {"start": 1818.5, "end": 1821.54, "text": " Like input volume so so this is the real image"}, {"start": 1821.54, "end": 1825.94, "text": " It has three channels and have certain spatial dimension and this this is the up sampled"}, {"start": 1826.42, "end": 1829.0600000000002, "text": " So we first down sample and then up sample the image"}, {"start": 1829.78, "end": 1832.3400000000001, "text": " In order to mimic the process here, okay"}, {"start": 1832.34, "end": 1836.58, "text": " So what I say here is that they are just kind of blurring and"}, {"start": 1837.3, "end": 1839.3, "text": " Basically, it should not be"}, {"start": 1839.86, "end": 1842.58, "text": " So the way they are probably approaching this is"}, {"start": 1843.22, "end": 1849.86, "text": " If you do like take an image of the full size and then you don't sample it and then you up sample it again"}, {"start": 1850.4199999999998, "end": 1854.74, "text": " I'm fairly sure that if you go into the spectral domain"}, {"start": 1855.3, "end": 1860.98, "text": " You can you can just devise a kernel a certain kernel like a kernel like this"}, {"start": 1860.98, "end": 1864.42, "text": " Like a kernel like in the sense in convolutional neural networks"}, {"start": 1864.58, "end": 1871.38, "text": " You can devise a kernel that's gonna modify the specter spectrum of this of this image in such a way that we get a blurred"}, {"start": 1871.54, "end": 1876.18, "text": " We get the same effect as if we were to down sample explicitly and up sample again"}, {"start": 1876.42, "end": 1883.46, "text": " But that's just a implementation detail that would probably make this process a bit more like efficient and optimized"}, {"start": 1885.14, "end": 1890.18, "text": " Okay, so the terminology I was I was trying to to get at was the dual dual discrimination"}, {"start": 1890.18, "end": 1896.02, "text": " That's how they call it and it not only encourages the final output to match the distribution of real images"}, {"start": 1896.42, "end": 1904.5, "text": " But also offers additional effects. So it encourages the neural rendering to match the distribution of thousand sampled real images"}, {"start": 1904.8200000000002, "end": 1910.5, "text": " So that's remember why so neural rendering to match the distribution of down sampled real images"}, {"start": 1910.74, "end": 1916.26, "text": " So that's because as you can see here, so this is the neural rendered image. So this thing here"}, {"start": 1916.26, "end": 1923.86, "text": " So as you recall we basically took the first three channels and we get this and now once we up sample this"}, {"start": 1924.58, "end": 1931.62, "text": " We are basically making sure that this thing looks as close as this real real image that was down sampled and up sampled"}, {"start": 1931.86, "end": 1933.86, "text": " Okay, so that's why we we have this"}, {"start": 1934.8799999999999, "end": 1936.8799999999999, "text": " consistency constraint"}, {"start": 1937.14, "end": 1938.74, "text": " um, and"}, {"start": 1938.74, "end": 1943.3, "text": " It encourages the super resolved images to be consistent with the neural rendering"}, {"start": 1943.3, "end": 1951.06, "text": " And what they mean by that is that basically this image here is going to be encouraged to be looked the same as this image here"}, {"start": 1951.06, "end": 1954.5, "text": " Which means that the super resolution is going to do its job"}, {"start": 1954.5, "end": 1961.86, "text": " Basically, it's going to just up sample the image without modifying the pose or the appearance or whatnot of this of this of this output image. Okay"}, {"start": 1962.58, "end": 1968.26, "text": " So that's it. Let's now continue and let's see some results. Results are fairly stunning"}, {"start": 1968.26, "end": 1974.04, "text": " You can see the identity of this of this uh, a man looking uh frontally"}, {"start": 1974.34, "end": 1981.78, "text": " And then when we change the pose the you can still say you can still tell that this is the same person and similarly here"}, {"start": 1982.26, "end": 1987.54, "text": " So this thing with the eyes is a bit creepy. It doesn't look like this person should be watching at the camera"}, {"start": 1988.1, "end": 1992.34, "text": " Looking at a camera. It looks like the eyes should be going there or something like that. But yeah"}, {"start": 1992.98, "end": 1995.94, "text": " Looks kind of weird. We're gonna see why that is a bit later"}, {"start": 1995.94, "end": 2003.3, "text": " Um, basically there are some problems where where the underlying geometry of the eye socket is concave instead of convex"}, {"start": 2003.3, "end": 2005.3, "text": " Which is creepy. Um"}, {"start": 2005.78, "end": 2012.3400000000001, "text": " They compare with the baseline models and as you can see every one of these has some problems either the geometry is super"}, {"start": 2012.98, "end": 2017.38, "text": " Bad or it's super super like very like smooth"}, {"start": 2018.1000000000001, "end": 2023.8600000000001, "text": " Um, and here it's also you can see it's the the image does not look realistic the the tooth"}, {"start": 2023.86, "end": 2030.8999999999999, "text": " The teeth here are are smoothened out and the geometry as well. And here is the the this method the proposed method looks way better"}, {"start": 2030.8999999999999, "end": 2035.06, "text": " Compared to the baselines just qualitatively analyzing these images"}, {"start": 2036.58, "end": 2037.3799999999999, "text": " Um"}, {"start": 2037.3799999999999, "end": 2042.1, "text": " Quantitative analysis show the same thing comparing with these baselines looking at a bunch of different metrics"}, {"start": 2042.1, "end": 2047.2199999999998, "text": " I'm not gonna get into what these metrics are fid basically a fresh raise inception score"}, {"start": 2047.22, "end": 2056.1, "text": " Um, uh, like depth and they're just comparing geometry and the quality of the uh, generating images and you can see that the uh, the the"}, {"start": 2057.06, "end": 2064.02, "text": " This approach is way better compared to the baselines. They also compare the speed the number of frames per second and you can see that"}, {"start": 2064.58, "end": 2069.86, "text": " It's lagging a little bit behind g-roff and some of those methods, but as you saw the quality is way better"}, {"start": 2069.86, "end": 2074.66, "text": " So that's kind of decent trade-off finally. They they use this fax smile. Um,"}, {"start": 2074.66, "end": 2079.46, "text": " Like sdd, I guess it's a standard deviation. I think this just kind of shows that"}, {"start": 2079.94, "end": 2081.2999999999997, "text": " the the"}, {"start": 2081.2999999999997, "end": 2086.02, "text": " Multi-view consistency is better when you use the dual discrimination and the"}, {"start": 2086.8199999999997, "end": 2089.7, "text": " Adding the generator pose conditioning the gpc"}, {"start": 2090.2599999999998, "end": 2092.2599999999998, "text": " to the model, so that's the"}, {"start": 2092.8199999999997, "end": 2093.8599999999997, "text": " basically"}, {"start": 2093.8599999999997, "end": 2100.2599999999998, "text": " The the camera parameters that we were feeding into generator to condition it. So that was this part here"}, {"start": 2100.26, "end": 2106.42, "text": " So using this conditioning on the parameters the generator the mapping network part of the generator"}, {"start": 2106.98, "end": 2108.98, "text": " That helps a lot"}, {"start": 2109.38, "end": 2111.38, "text": " incur this this"}, {"start": 2112.0200000000004, "end": 2114.0200000000004, "text": " This consistency, okay"}, {"start": 2114.34, "end": 2116.34, "text": " Let's continue and"}, {"start": 2116.5800000000004, "end": 2120.9, "text": " See some results. So this is i'm going to skip this this is from the style again. This is very"}, {"start": 2121.5400000000004, "end": 2124.82, "text": " Already very very clear from the style again papers"}, {"start": 2124.82, "end": 2131.86, "text": " You can mix up the style from the row. So this person here dictates the style and these persons in the column"}, {"start": 2132.1800000000003, "end": 2136.6600000000003, "text": " Dictate the general geometry and appearance you can see they're kind of mixing up this this kid here"}, {"start": 2136.6600000000003, "end": 2142.7400000000002, "text": " You can see how the style of the kid the style of the kid is changing depending on the leftmost column. Okay"}, {"start": 2144.5, "end": 2146.5, "text": " I thought explaining how this"}, {"start": 2146.82, "end": 2152.7400000000002, "text": " Pti works in more details, but it's gonna take it's gonna make it a little bit more complicated"}, {"start": 2152.74, "end": 2155.9399999999996, "text": " Details, but it's gonna take it's gonna make this video too long"}, {"start": 2156.3399999999997, "end": 2159.22, "text": " So I can maybe cover this separately in one of the next videos"}, {"start": 2159.9399999999996, "end": 2163.4599999999996, "text": " But basically what I managed to do is given the image"}, {"start": 2164.18, "end": 2167.9399999999996, "text": " The desired target image they managed to find the latent"}, {"start": 2168.4199999999996, "end": 2175.22, "text": " That generates that that image and then they can basically modify and edit the image"}, {"start": 2175.8399999999997, "end": 2177.22, "text": " semantically"}, {"start": 2177.22, "end": 2179.9399999999996, "text": " Inside of that latent space. So that's very very cool"}, {"start": 2179.94, "end": 2187.38, "text": " So what you can notice here I mentioned the concaveness of the eyes and that's the reason why it always looks like the eyes are looking"}, {"start": 2187.78, "end": 2190.42, "text": " directly into us into yeah"}, {"start": 2191.46, "end": 2196.66, "text": " Directly into our eyes, whatever is because you can see that the eye sockets are kind of concave here"}, {"start": 2197.62, "end": 2201.06, "text": " I think they they mentioned that somewhere down here. Let me let me find it"}, {"start": 2201.3, "end": 2205.94, "text": " So furthermore ambiguities that can be explained by geometry remain unresolved"}, {"start": 2205.94, "end": 2213.8, "text": " For example by creating concave eye sockets the generator creates the illusion of eyes that follow the camera an incorrect interpretation"}, {"start": 2214.18, "end": 2218.9, "text": " Though the renderings are view consistent and reflect the underlying geometry. Okay"}, {"start": 2219.78, "end": 2223.86, "text": " I'm not gonna dig deeper into that but um, i'm just gonna"}, {"start": 2225.38, "end": 2230.26, "text": " First address this paragraph here because I think it's also important not that like related to this concrete paper"}, {"start": 2230.26, "end": 2235.62, "text": " But I think it's a in general. I'm just yeah, I have this are these paragraphs useful"}, {"start": 2235.62, "end": 2239.22, "text": " Uh asking myself whether this is useful i'm gonna read it through"}, {"start": 2240.02, "end": 2242.18, "text": " Read it out loud so you can you can judge yourself"}, {"start": 2242.58, "end": 2248.5, "text": " So the single view 3d reconstruction or style mixing applications could be misused"}, {"start": 2248.9, "end": 2256.58, "text": " So it could be misused for generating edited imagery of real people. Okay, we are all familiar with deep fakes with what the GANs are in general"}, {"start": 2257.22, "end": 2258.5, "text": " kind of"}, {"start": 2258.5, "end": 2263.94, "text": " Like a problematic in that sense such misuse of image synthesis techniques poses a societal threat"}, {"start": 2263.94, "end": 2271.14, "text": " And we do not condone using our work with the intent of spreading misinformation or tarnishing reputation. I mean"}, {"start": 2271.94, "end": 2279.08, "text": " That's also kind of should be obvious authors do not want their paper their work to be used maliciously"}, {"start": 2280.1, "end": 2283.4, "text": " I don't see any added information in these sentences"}, {"start": 2283.94, "end": 2290.66, "text": " And finally, we also recognize a potential lack of diversity in our faces results stemming from implicit biases of the data sets"}, {"start": 2290.66, "end": 2298.66, "text": " We process and this like this same sentence could be copy pasted into so many different papers like"}, {"start": 2299.54, "end": 2305.48, "text": " Let's take let's replace this 3d GAN paper. Let's replace it with uh, like a gpt3"}, {"start": 2306.8199999999997, "end": 2312.02, "text": " Or whatever your favorite language model is now in 2021 maybe without"}, {"start": 2312.64, "end": 2314.58, "text": " 2.0 or something. I don't know"}, {"start": 2314.58, "end": 2321.7799999999997, "text": " And basically so it's also can be misused and the author and secondly the authors condone using their work maliciously"}, {"start": 2321.86, "end": 2327.2999999999997, "text": " so that's also true and it also is true that depending on the biases in the"}, {"start": 2328.1, "end": 2330.1, "text": " they basically use the the the"}, {"start": 2330.66, "end": 2335.86, "text": " Text data set that was collected from the internet and reddit which has a lot of biases for like"}, {"start": 2336.42, "end": 2338.42, "text": " various various things like uh"}, {"start": 2338.66, "end": 2343.22, "text": " For example muslims being associated with crim like with terrorism stuff like that"}, {"start": 2343.22, "end": 2347.4599999999996, "text": " So some bad societal biases. So the point is"}, {"start": 2349.06, "end": 2353.8599999999997, "text": " Whether we should just be copy pasting this everywhere because this can be applied for every single paper"}, {"start": 2353.98, "end": 2356.74, "text": " Yeah, I don't mean anything bad by by saying all of this"}, {"start": 2356.74, "end": 2362.02, "text": " I just want to kind of raise the concern of whether this is going to be of any practical use paragraphs such as this one"}, {"start": 2362.02, "end": 2366.66, "text": " I'm not sure. I'm not smart here. I'm just I'm just uh, yeah speaking out loud here"}, {"start": 2366.66, "end": 2372.92, "text": " Um, finally, let me just find one additional thing here in the appendix"}, {"start": 2373.7, "end": 2376.12, "text": " Um, it has to do with the intrinsics"}, {"start": 2377.54, "end": 2379.54, "text": " Okay, here it is"}, {"start": 2379.54, "end": 2381.54, "text": " They say that we assume"}, {"start": 2381.62, "end": 2387.62, "text": " Fixed so the for the ffhq data set which is the data set of those high resolution human faces"}, {"start": 2388.74, "end": 2391.7, "text": " Not the cats the cats are the this is the cat's data set"}, {"start": 2391.7, "end": 2399.14, "text": " So they say the via zoom fixed camera intrinsics across the entire data set with a focal length of four point twenty six"}, {"start": 2399.2999999999997, "end": 2405.54, "text": " Times the image width equivalent to standard portrait lens. So that means that that's kind of"}, {"start": 2406.5, "end": 2407.7799999999997, "text": " assumed"}, {"start": 2407.7799999999997, "end": 2411.7799999999997, "text": " So that means it might not be correct. I'm not sure how the data set was collected"}, {"start": 2412.02, "end": 2415.2999999999997, "text": " So it may happen that different images were collected with different cameras"}, {"start": 2415.2999999999997, "end": 2418.4199999999996, "text": " which means this would not hold as a as a hypothesis and"}, {"start": 2418.42, "end": 2422.42, "text": " And they are using certain off-the-shelf detectors to figure out the extrinsics"}, {"start": 2422.42, "end": 2428.1800000000003, "text": " So I haven't seen not sure I may be be making a mistake here, but I haven't seen any"}, {"start": 2429.06, "end": 2435.06, "text": " Ablation on how on how robust the pipeline is to the errors in the this and this"}, {"start": 2435.06, "end": 2440.1800000000003, "text": " If this turns out not to be the case that this is a correct intrinsics for this data set"}, {"start": 2440.9, "end": 2442.9, "text": " Yeah, I just thought kind of mentioning that"}, {"start": 2444.9, "end": 2447.54, "text": " Also, one thing I'm not completely sure is how their"}, {"start": 2447.54, "end": 2453.38, "text": " Generating these geometry visualizations. I can assume that let me just go back to the pipeline here"}, {"start": 2453.86, "end": 2456.9, "text": " It's going to be easier to explain what I mean"}, {"start": 2459.7, "end": 2466.98, "text": " Okay, it's a long long paper, okay, what is probably happening is that they are using the density information"}, {"start": 2468.5, "end": 2472.9, "text": " So because so they pass array through the scene they have the density"}, {"start": 2472.9, "end": 2478.02, "text": " Information and they can kind of do a threshold and aggregate that density information"}, {"start": 2478.34, "end": 2480.5, "text": " So maybe these charts will be even better"}, {"start": 2480.5, "end": 2484.5, "text": " So they have something like this and then they can get just kind of threshold"}, {"start": 2485.2200000000003, "end": 2488.9, "text": " the the density and then use that to generate the"}, {"start": 2490.26, "end": 2496.26, "text": " Geometry, but I'm not sure whether I'm fairly sure that we have a lot of noise and glitching"}, {"start": 2496.26, "end": 2498.26, "text": " So it'd be nice to kind of"}, {"start": 2498.26, "end": 2502.98, "text": " Yeah for people who do not do this every day including me"}, {"start": 2502.98, "end": 2507.46, "text": " It'd be nice to have some hint of how stuff is done because yeah"}, {"start": 2508.5800000000004, "end": 2512.42, "text": " Understanding how geometry is actually visualized seems to be fairly important"}, {"start": 2512.42, "end": 2519.7000000000003, "text": " Anyways, if somebody knows exactly how this is done, feel free to comment down below and hopefully you like this video"}, {"start": 2520.82, "end": 2523.86, "text": " Let me know whether the lack of camera is"}, {"start": 2523.86, "end": 2529.7000000000003, "text": " Ruining the experience for you guys. So i'll make sure to in any case bring it back"}, {"start": 2530.26, "end": 2533.2200000000003, "text": " Very soon and create my youtube studio here in london"}, {"start": 2533.94, "end": 2536.6600000000003, "text": " Hopefully in like a couple of weeks in any case"}, {"start": 2537.46, "end": 2544.34, "text": " Subscribe if you if you like this video share out the the video with friends and join the discord community"}, {"start": 2544.34, "end": 2550.26, "text": " Because there is a lot of awesome cool people there which are willing to help each other out. So"}, {"start": 2550.26, "end": 2555.32, "text": " So consider joining the the community there and until next time. Bye. Bye"}]