Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation Paper Explained
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover the "Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation" paper where the authors devised a novel representation that made the robotic manipulation much more robust and capable of generalizing to novel shape instantiations and arbitrary unseen poses.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2112.05124
✅ Website: https://yilundu.github.io/ndf/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
02:00 Neural Point Descriptor Field
12:30 Introducing SE(3) equivariance
22:35 Inducing energy landscape and optimization
25:10 Generalizing the neural descriptors to a pose
28:00 Visualizations, Results and Ablation
29:55 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Kulsoom Abdullah
Petar Veličković
Bartłomiej Danek

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#robots #equivariance #neuralrepresentations
 What's cracking guys? Finally I'm making a new video. Basically I'm still setting up in London, so I don't have my camera set up ready. And yeah, so this video is going to be a bit different compared to all of the other research paper overviews I've done so far. And in any case, I'm going to cover this paper called Neural Descriptor Fields, SE3 Equivariant Object Representations for Manipulation by these people from MIT, Google Research, and University of Toronto. But before I even start digging into the paper and explaining the method, let's see the video demo they prepared. Okay, here what we can see is that they'll make a couple of demonstrations, and based on those demonstrations, where the monks and in general whatever object they're using are upright, so that's kind of simple. And here you can see during the test time, the pose can be arbitrary as well as the instance doesn't have to be the same as the ones used in the demonstration. So they're of the same class, like for example, like a glass or whatnot, but basically the pose and the actual instance can and will be different. And you can see the awesome generalization that the method is showing. So things to keep in mind is that only this grabbing and placing the object, the glass in this case, on this thing is what the interesting part of this method is. The actual intermediate moving is just done via off-the-shelf methods. So yeah, and you can see here that even with different more complex objects, you can see a lot of variance between the objects of the same class, and it's still managing to work correctly. Okay, getting back to the paper, let's see how this thing actually works. As I said, the interesting part, generalization to like grabbing the object and placing the object. Everything in between, so this part here is handled by certain heuristics. Let me just find that part. They mentioned it somewhere here. We rely on off-the-shelf inverse kinematics and motion planning algorithms to execute the final predicted pick and place task. So let's focus on the actual components that enable this awesome generalization. So let's start here. So the key idea is to represent an object as a function f that maps a 3D coordinate x to a spatial descriptor z, which is a function of x of that 3D coordinate. So we want to somehow, and they additionally say here, that f may further be conditioned on an object point cloud P, which has, as you can see, it's a 3D points, and there are n number of those points in a point cloud. And the idea is to output category level descriptors f of x conditioned on the point cloud P. So the reason it's called neural point descriptor fields. So neural is going to be clear in a couple of seconds. Basically, they're using a neural network to learn these representations. Point is because we are currently just inputting the point here, I guess. Yeah. And descriptor fields because we are associating with every 3D point, we are associating a vector, hence it's a field. And finally, descriptor is because we'll be describing the object that we want to grab in the scene like this. So we are, so this is very abstract definition. We still don't know how to actually implement this, but let's slowly move on and we'll understand this better in a second. So blah blah mesh editor at all represent a 3D shape as an MLP. So it's a multilayer perception phi that maps a 3D coordinate x to its occupancy value. So what the idea here is, so these are called neural implicit representations. And the idea is that the weights of the neural network actually learn like basically the structure and the position of the object. So you basically input a point x and this function phi will tell you whether the point x lies inside of the object, so that will be quantized, that will be quantified by 1, or it lies outside of the object when we'd have 0 as the output. And now you can imagine if you were to systematically kind of do a sweep over various x's over certain cube, something like this, you'd basically get, let me just draw it here. Yeah, by the way I'm drawing using touchpad, so it's really hard. So what you'll get here is because the phi will output 1 or 0, basically you're going to, so basically this way you can render, let's call it render an object. So that means that this like neural network phi learned how to represent an object implicitly. And we can make that explicit by, as I said, doing some sweep over the input points x. Finally, so this as you can see here, if we were to do it like this, it would be similar to NURF and we'd be overfitting to one specific shape. And we want to be able to generalize to various instances and various classes as well. So they say here we're interested in learning a low dimensional latent space of 3D shapes. And that's basically what they mean by that is that we want to encode, we want to learn how to take an object, the point cloud of an object, we want to pass it to some point cloud encoder and we're going to see they're using point net to do this and we get a latent representation here. So the point of this representation is to make, to just like abstract away the unnecessary details. Like we want to make sure that we understand that this is a mug or whatnot. Like even if the mug was way like higher, taller, like here and maybe the handle was like here instead of on top of the mug, we want to be able to kind of represent these two as the same latent feature vector here. So that would be cool. Okay, so let me go back here. So as I said, they're interested in learning those low dimensional latent space like basically features. And these latent codes are obtained as an output of point net. I already mentioned that based point cloud encoder epsilon that takes as input a point cloud P leading to a conditional occupancy function here. So as you can see here, now we have, so this formula here directly corresponds to this block here and we input like a point, we input like a feature vector representing like an abstract representation of our input shape. And you can see this is still an occupancy function. So we get zero if we are inside of this specific shape and we get one, so we get zero if we are outside and one if we are inside of that particular shape, given the point X. And finally they say here the full model can be trained end to end on a data set of partial point clouds in corresponding occupancy voxel grids of the object's full 3D geometry, thus learning to predict the occupancy of a complete 3D object from a partial point cloud. So a cool thing about this is that you can create a synthetic data set and basically because it's synthetic you can kind of corrupt the point clouds, make partial point clouds, but you still have the ground truth of voxel grids like non-corrupted. And that means that you'll be able to train this whole thing using partial point clouds, but the occupancy, so this output here, so zero or one, is still going to be correct. So later on when we have real robot, all of the mugs as you can see above here, so basically you can imagine that the depth cameras which are using in the scene here, they've positioned them somewhere outside of this picture, will have only a partial, will be able to retrieve a partial point cloud. And I mean that's why it's a smart idea to train the NDF, this neural point descriptor in the same manner. Okay, so why is this important? Why are we training this neural network to predict the occupancy function? So it turns out there are some nice properties that once we train the model the way I just described, so we are inputting partial point clouds and we're trying to basically output zero or one depending on whether this point x is inside of the shape or not. And we do have the ground truth data because remember we're using synthetics, we can easily use synthetic data to train this model. So now let's see what the interesting part and properties of this neural point descriptor field network FR. So, this is here, to enable category level object manipulation, a spatial descriptor for a coordinate x given a point cloud P should encode information about the spatial relationship of x to the salient features of the object. Our key insight is that the category level 3D reconstruction objective trains this network PHY, so that's the occupancy network, to be a hierarchical course defined feature extractor that encodes exactly this information, so the spatial relationship of x to the salient features of the object. Basically PHY is a classifier whose decision boundaries is the surface of the object, so that's how it was trained. But like in a nutshell what they just said here is that they can use the representations that were learned during the training of this occupancy function PHY and use those as a way to capture these relationships of the point x to salient, geometrically salient parts of this shape. Ideally what would happen is that these descriptors would be the same. So if I had a point x here, let me change the color here to something else, like if I had a point here next to this handle of this small mug and if I had a point here next to this handle of this tall mug, ideally the descriptors would be the same and that's what's going to enable the ultimate generalization that the authors showed. So as I said their actual representation is going to be a concatenation of these representations here and they formulaically capture that exactly here. So we thus propose to parametrize our neural point descriptor field F as the function that maps every 3D coordinate x to the vector of concatenated activations of PHY. So basically, let me change the color, this is just a symbol for concatenation, we iterate through all of the layers from 1 through L, we grab the intermediate representations of the occupancy function of PHY and the notation is a bit sloppy here because we are not inputting point and the latent vector of the shape in every single of these layers, we are just inputting these in the first input layer. It's a bit sloppy but you get the point. Okay, so let me now explain, let me just show you one visualization, it's going to be a bit easier. So as I said, ideally what would happen is that even if we had different mugs as you can see here and we have these green points, this is the x, ideally we would have the same descriptor. So that's the whole point, that's the idea here. And now there is a problem with this current formulation and the thing we just have done. Basically, let me just kind of give you some understanding of this encoder here. It's a point net from a paper that I think is from 2015 or something. It's a very interesting architecture but the main idea of this architecture is the following. The main idea is the following, you want to have this permutation invariant property. So what the point net did is you input this array of points, so n times 3 because we have 3D points, we have n of those. What we do is we use these shared MLPs and some transforms and we basically independently process each of these points. So this point here is going to be independently processed from all of the other ones, at least in this layer. Then we have the featuring transforms but the most important part is basically here and because we do a global pooling exactly here, that means that this final representation will be the same no matter how we permute this input here. So that means that point net is permutation invariant. But now imagine what happens if I were to translate. So imagine we had a single point, imagine we had this shape like this mug here. And imagine what would happen if I were to translate that mug right here. What would happen is that all of the points in this point cloud would be basically modified such that each point would be, let's say, we had to add the translation vector t to those. And now once you pass those here, you can imagine that the representation will change. Okay, and the same thing would happen if I were to rotate the mug. We have a different representation. And that's all fine when we have a standalone point net. But what I'm getting at here is that we want to make sure that if I were to rotate this mug here, let's say we have a point x here. And if I were to rotate this mug, however we want, for the sake of simplicity, I'm just going to reduce this problem to a 2D problem. So let's say we basically have a 2D square and we have a point x here. Now if I were to rotate this square, so let's put some coordinate system here to make it a bit more clear. Basically, if, now let's draw the coordinate system again. And if we were to rotate the square for like 45 degrees or whatnot and rotate the point x the same way, so basically you'd want to ideally have the same descriptor for these two, right? Because their relative relationship has not changed. And with the current implementation, remember if we were to change the 3D points, so if we were to change x and if we were to change the rotation or if we were to translate this object, as I said, this latent vector here is going to change, x is going to change, so that means that the descriptor is definitely going to change. And in order to enable the generalization that they showed, it's very important to make this SE3 like an equivalent or just a fancy way of saying basically equivalent to translation and rotation. And they also, I think they're kind of being sloppy with the equivalent versus invariance. I've noticed that a couple of times, but yeah, hopefully you'll understand the gist of the paper. So they say here that in other words, we require f to be invariant to joint transformation of x and p. So again, f is, remember, the spatial descriptor, implying the descriptor field should be equivalent to SE3 transformation of p. So here they just formulaically represent that if we were to transform both x and the point cloud p with some rotation and some translation, we want to make sure that the output of the function f, remember, that's the descriptor, that's this whole thing is called f, and only the small portion here is called 5, that's the occupancy network. But we care about this descriptor, the neural point descriptor field, okay? So as I said, we want to make sure that those outputs are the same. They basically add two, let's call it tricks, to introduce this invariance or equivalence. The first thing is fairly simple. You basically first, if you want to be invariant to the translation, you can just find the mean point of the point cloud, so that's the mu here, and you subtract mu from both the input point x and from the point cloud p, and by doing this, you're guaranteed to always have this invariance. To understand why that is, and hopefully it's very, very simple, but let me try and kind of make it a bit clearer. If we have a coordinate system here, and I'm again going to be in a 2D space because it's harder to draw 3D objects using touchpad. So if we had, let's say we have a square here and a point here, and let's say we have a translated version of this same square, so I'm just going to use a different color here, so let's say we have it here, okay? So if we were to find the mean point, which would be here for this object, and it would be basically here for this square, for the red square, and if we were to subtract those from the corresponding two cases we have, we'd end up with the same result both times, so we'd basically end up with this configuration, if we were to subtract this vector, and we would end up with the same configuration, if we were to basically subtract the other vector. And because of that, the inputs are the same, and then obviously we do that even before we pass the data through the function f, and that's why we're going to be trivially have this property satisfied. As for the rotation, it's not that simple. You can do the same approach and kind of guarantee that you have invariance, because imagine if I were to kind of rotate this thing here, and now if we try to kind of just subtract the vector, we'd end up with a square that looks something like this, which obviously will be different input, and then f will just produce different output, so we don't have the invariance property. So to achieve the rotation, active variance, they say here we rely on recently proposed vector neurons, which propose a network architecture that equips an occupancy network, i.e. the composition of epsilon, which is the point net encoder, and phi, which is basically the small submodule in the occupancy network, with full SO3 active variance. So SO just means rotation invariance, active variance, whereas the SE3 means both translation and rotation, just keep that in mind. And formulaically that means we want to achieve this, we want to achieve the same output, even if we were to transform the input point X and the point cloud P using some arbitrary rotation matrix R. So how they achieve that is basically described in this paper called Vector Neurons. I'm going to just give a super high level explanation here. Basically what they've done is instead of using scalar neurons, so this is your classical linear layer in an MLP, they instead have as an output not a scalar but a vector, so that's a generalization of these scalar neurons. And similarly, the implementation, this is, if you can guess, this is an implementation of a ReLU non-linearity in these vector neural networks, and basically you may be able to guess why this is, but in a scalar case you have something like this, so you have some input X scalar, you have output Y, and basically ReLU will kind of saturate this to zero, so this is zero, and then we're going to have for positive X, we're going to have a simple linear function, so Y equals X. So here we have a similar, this is just a generalization of ReLU to 3D space. Basically if we have vector Q, nothing will happen, so that's the same as if we were to have input, so let me just for clarity name this Q as well. So in this case Q is a scalar, so in the case where Q is in the positive part of the domain here, Q remains unchanged. Basically Q, because this is Y equals X, we're going to have the output will also be Q, and that's what we get here. But in the other case where the Q lies in the, let's call it negative half space, which would be equivalent to this part here, we basically project the vector and we just keep the component that's alongside this plane, and we drop the component that's alongside negative K direction. So yeah, it would take a whole video to explain why this works and why this introduces SO3 equivariance, but let's just take it as a theory and it just works. So now we have all of the components necessary to understand how that robot is gripping those mugs, even if the pose and even if the shape, the actual instantiation of that mug is changing. The idea lies in creating, inducing these energy landscapes, and we are using descriptors in the following manner. Okay, let me use their visualization because it's going to be easier. So imagine we have basically, this is P hat is, you can treat this as a demonstration, a mug that was used during demonstration, and P is the one used during the test time. So imagine we have this point X here in green and we have the shape here, and basically we can pass those and we get out, as the output we get the description, the descriptor of their mutual relationship. Let's put it that way. On the other hand, we have novel point cloud P here. It's, as you can see, a bit differently shaped. It's not the same as the one above. And now the idea is to find corresponding, what will be the corresponding point to this X for this current configuration here? And the answer you would give intuitively is it should be somewhere here, right? And the reason it's going to work is because, as you can remember, how we train the F function, or neural descriptor, is such that this is what the energy landscape is going to look like. So basically for points which are close to the handle here, because remember the shape is going to be abstracted away because of the point net and the way we trained the whole system, and that's going to cause the energy landscape to look like this, which means once we use simple optimization, so we'll be tweaking, so what will happen is we'll be using Adam, so they've been using, in practice they've been using Adam, so we'll be tweaking the parameters of X, so basically three scalars, we're just tweaking those such that point X will be moving towards this low energy part of the landscape, and once that happens, basically this, so yeah, this is this loss, we are minimizing that, and as we are minimizing that, we're basically finding this point that we just found intuitively a couple moments ago. And because of that, the robot will basically know how to grip this handle. There will be like a simple high level explanation of how the thing works. So now the problem is with this thing is that the grip of a robot actually is not a single point, because if you have a single point, you can have multiple frames, coordinate frames attached, you can have infinite coordinate frames attached to that point, so they just kind of generalize this to the case where we want to have the pose information as well, and how they represent the pose is very similar, you just basically concatenate, so you take all of the points here, so this is again concatenation symbol, you take all of the points from this query point cloud, so those are these points with various colors as you can see here, and you just basically concatenate their descriptions, and that's how you get the final description for the pose. Once you have that, by the way, in order to get those points, they mentioned here that you're using a robust heuristic to sample points uniformly at random from within the bounding box of the rigidbody S, whereas you can think of S as like a gripper in the case of picking up an object, and in the case when you want to place it, it's going to be some other object, and in the case when you want to place the actual shape onto some table or whatnot, onto a shelf, you won't have a gripper, you have some different salient object, and basically what they do is, as you can see here, finally they basically do K demonstrations, so 1 through K, and they collect these descriptors, so for where the gripper was during picking and where the placement surface was when the object was placed in the demonstration, and then they just average out all of these descriptors, and they end up with a tuple, basically this Z pick and Z rel with these bars on top of them, that's a kind of silly notation, but yeah. And as I said, once you have the descriptors, you can just minimize this energy landscape to get the position for the novel object in a novel instance of the same class with a different orientation or whatnot. And again, they mention here, we rely on off-the-shelf inverse kinematics and motion planning algorithms to execute the final predicted pick and place task, so that means that the whole trajectory between the two endpoints is not something they were trying to solve in this paper. Yeah, here's just some visualizations of how you start up with some random points, and then because the demonstration was such, and then that means we had certain descriptor as an output of this demonstration, now we just minimize the loss, so we move towards the lower parts of the energy landscape, and we end up with the same, as you can see, analogous position for this novel shape, so basically this test shape is potentially different than the demonstration one. Results are, surprise, surprise, way better compared to some of the baselines, especially when you're using upright pose, the baseline is kind of on-pair, let's call it on-pair, but its NDF is still better compared to DON, but like when you start using arbitrary poses, that's when this thing starts shining, and it's way better compared, as you can see the gap increases significantly here and here. Also, you can notice that there is some drop in the performance here, and I think that they mentioned that the main reason was that the point clouds kind of changed, because during the demonstrations the robots only saw the mug or whatever object they were using from the bird view perspective, and when they use arbitrary pose, obviously there are some novel occlusions and these occlusions that happen, and so that's why there is some degradation there. That's pretty much it. Here they just show some ablations, basically it's better to use, like should we use final layer or the first layer of the occupancy network or all layers, and we saw that they ended up using all of these layers as a representation, and that turned out to be the best option, and yeah, that's pretty much it. There is a couple of things they also mentioned here. Yeah, they also showed that as the number of demonstrations increases, you can see that the performance is increasing, so it's important that they use around 10 demonstrations so that this can work, so this is a few-shot learning, let's say, and as a point of further potential research, they say that further NDFs can only define transferable energy landscapes over poses and points. Future work may explore integrating such energy functions with trajectory optimization to enable NDFs to transfer to full trajectory. So that's the part I said that the actual movement between the picking up and placing is done using these off-the-shelf methods. So I love the idea. There are things that, in my opinion, should be improved upon, obviously, not using the heuristics, not using this off-the-shelf algorithms to plan the trajectory, but maybe inducing an energy loss scape across this whole thing. Also, I'm always trying to think when I see a method like this, whether our brain is doing something similar. It's not that we have to copycat our brain, but we need to find some inspiration there, and it's kind of the only current implementation of AGI that we have present in the world, so I guess it's a good starting point. But just kind of doing gradient descent on the energy landscape doesn't seem like the thing that our brain is doing. But when I saw the demos they showed, it's quite impressive, and basically, yeah, kudos to the authors. Some parts of the paper could have been written a little bit more clear. Yeah, at times it's a bit hard to understand, and the 10-pager format for the papers is not helping that much because people have to squeeze in stuff, and then you lose the structuring of the documents. It's hard to read, and yeah. That said, hope you like this video. As I said, it's a bit different because I still don't have my setup here in London, but I'll be continuing with the Jack series. Next up, you can expect my Flex video, Haiku video, and then I'll start doing some videos on ML Ops, and yeah, stay tuned for that. Until next time, bye-bye.
[{"start": 0.0, "end": 10.5, "text": " What's cracking guys? Finally I'm making a new video. Basically I'm still setting up in London, so I don't have my camera set up ready."}, {"start": 10.5, "end": 17.0, "text": " And yeah, so this video is going to be a bit different compared to all of the other research paper overviews I've done so far."}, {"start": 17.0, "end": 37.0, "text": " And in any case, I'm going to cover this paper called Neural Descriptor Fields, SE3 Equivariant Object Representations for Manipulation by these people from MIT, Google Research, and University of Toronto."}, {"start": 37.0, "end": 44.0, "text": " But before I even start digging into the paper and explaining the method, let's see the video demo they prepared."}, {"start": 44.0, "end": 59.0, "text": " Okay, here what we can see is that they'll make a couple of demonstrations, and based on those demonstrations, where the monks and in general whatever object they're using are upright, so that's kind of simple."}, {"start": 59.0, "end": 68.0, "text": " And here you can see during the test time, the pose can be arbitrary as well as the instance doesn't have to be the same as the ones used in the demonstration."}, {"start": 68.0, "end": 79.0, "text": " So they're of the same class, like for example, like a glass or whatnot, but basically the pose and the actual instance can and will be different."}, {"start": 79.0, "end": 84.0, "text": " And you can see the awesome generalization that the method is showing."}, {"start": 84.0, "end": 101.0, "text": " So things to keep in mind is that only this grabbing and placing the object, the glass in this case, on this thing is what the interesting part of this method is."}, {"start": 101.0, "end": 108.0, "text": " The actual intermediate moving is just done via off-the-shelf methods."}, {"start": 108.0, "end": 119.0, "text": " So yeah, and you can see here that even with different more complex objects, you can see a lot of variance between the objects of the same class, and it's still managing to work correctly."}, {"start": 119.0, "end": 124.0, "text": " Okay, getting back to the paper, let's see how this thing actually works."}, {"start": 124.0, "end": 132.0, "text": " As I said, the interesting part, generalization to like grabbing the object and placing the object."}, {"start": 132.0, "end": 140.0, "text": " Everything in between, so this part here is handled by certain heuristics. Let me just find that part. They mentioned it somewhere here."}, {"start": 140.0, "end": 150.0, "text": " We rely on off-the-shelf inverse kinematics and motion planning algorithms to execute the final predicted pick and place task."}, {"start": 150.0, "end": 155.0, "text": " So let's focus on the actual components that enable this awesome generalization."}, {"start": 155.0, "end": 169.0, "text": " So let's start here. So the key idea is to represent an object as a function f that maps a 3D coordinate x to a spatial descriptor z, which is a function of x of that 3D coordinate."}, {"start": 169.0, "end": 183.0, "text": " So we want to somehow, and they additionally say here, that f may further be conditioned on an object point cloud P, which has, as you can see, it's a 3D points, and there are n number of those points in a point cloud."}, {"start": 183.0, "end": 190.0, "text": " And the idea is to output category level descriptors f of x conditioned on the point cloud P."}, {"start": 190.0, "end": 198.0, "text": " So the reason it's called neural point descriptor fields. So neural is going to be clear in a couple of seconds."}, {"start": 198.0, "end": 207.0, "text": " Basically, they're using a neural network to learn these representations. Point is because we are currently just inputting the point here, I guess."}, {"start": 207.0, "end": 215.0, "text": " Yeah. And descriptor fields because we are associating with every 3D point, we are associating a vector, hence it's a field."}, {"start": 215.0, "end": 224.0, "text": " And finally, descriptor is because we'll be describing the object that we want to grab in the scene like this."}, {"start": 224.0, "end": 233.0, "text": " So we are, so this is very abstract definition. We still don't know how to actually implement this, but let's slowly move on and we'll understand this better in a second."}, {"start": 233.0, "end": 246.0, "text": " So blah blah mesh editor at all represent a 3D shape as an MLP. So it's a multilayer perception phi that maps a 3D coordinate x to its occupancy value."}, {"start": 246.0, "end": 251.0, "text": " So what the idea here is, so these are called neural implicit representations."}, {"start": 251.0, "end": 260.0, "text": " And the idea is that the weights of the neural network actually learn like basically the structure and the position of the object."}, {"start": 260.0, "end": 270.0, "text": " So you basically input a point x and this function phi will tell you whether the point x lies inside of the object,"}, {"start": 270.0, "end": 280.0, "text": " so that will be quantized, that will be quantified by 1, or it lies outside of the object when we'd have 0 as the output."}, {"start": 280.0, "end": 291.0, "text": " And now you can imagine if you were to systematically kind of do a sweep over various x's over certain cube, something like this,"}, {"start": 291.0, "end": 300.0, "text": " you'd basically get, let me just draw it here. Yeah, by the way I'm drawing using touchpad, so it's really hard."}, {"start": 300.0, "end": 315.0, "text": " So what you'll get here is because the phi will output 1 or 0, basically you're going to, so basically this way you can render, let's call it render an object."}, {"start": 315.0, "end": 330.0, "text": " So that means that this like neural network phi learned how to represent an object implicitly. And we can make that explicit by, as I said, doing some sweep over the input points x."}, {"start": 330.0, "end": 341.0, "text": " Finally, so this as you can see here, if we were to do it like this, it would be similar to NURF and we'd be overfitting to one specific shape."}, {"start": 341.0, "end": 348.0, "text": " And we want to be able to generalize to various instances and various classes as well."}, {"start": 348.0, "end": 353.0, "text": " So they say here we're interested in learning a low dimensional latent space of 3D shapes."}, {"start": 353.0, "end": 362.0, "text": " And that's basically what they mean by that is that we want to encode, we want to learn how to take an object, the point cloud of an object,"}, {"start": 362.0, "end": 371.0, "text": " we want to pass it to some point cloud encoder and we're going to see they're using point net to do this and we get a latent representation here."}, {"start": 371.0, "end": 377.0, "text": " So the point of this representation is to make, to just like abstract away the unnecessary details."}, {"start": 377.0, "end": 381.0, "text": " Like we want to make sure that we understand that this is a mug or whatnot."}, {"start": 381.0, "end": 399.0, "text": " Like even if the mug was way like higher, taller, like here and maybe the handle was like here instead of on top of the mug, we want to be able to kind of represent these two as the same latent feature vector here."}, {"start": 399.0, "end": 403.0, "text": " So that would be cool. Okay, so let me go back here."}, {"start": 403.0, "end": 411.0, "text": " So as I said, they're interested in learning those low dimensional latent space like basically features."}, {"start": 411.0, "end": 414.0, "text": " And these latent codes are obtained as an output of point net."}, {"start": 414.0, "end": 425.0, "text": " I already mentioned that based point cloud encoder epsilon that takes as input a point cloud P leading to a conditional occupancy function here."}, {"start": 425.0, "end": 443.0, "text": " So as you can see here, now we have, so this formula here directly corresponds to this block here and we input like a point, we input like a feature vector representing like an abstract representation of our input shape."}, {"start": 443.0, "end": 459.0, "text": " And you can see this is still an occupancy function. So we get zero if we are inside of this specific shape and we get one, so we get zero if we are outside and one if we are inside of that particular shape, given the point X."}, {"start": 459.0, "end": 474.0, "text": " And finally they say here the full model can be trained end to end on a data set of partial point clouds in corresponding occupancy voxel grids of the object's full 3D geometry, thus learning to predict the occupancy of a complete 3D object from a partial point cloud."}, {"start": 474.0, "end": 496.0, "text": " So a cool thing about this is that you can create a synthetic data set and basically because it's synthetic you can kind of corrupt the point clouds, make partial point clouds, but you still have the ground truth of voxel grids like non-corrupted."}, {"start": 496.0, "end": 506.0, "text": " And that means that you'll be able to train this whole thing using partial point clouds, but the occupancy, so this output here, so zero or one, is still going to be correct."}, {"start": 506.0, "end": 526.0, "text": " So later on when we have real robot, all of the mugs as you can see above here, so basically you can imagine that the depth cameras which are using in the scene here, they've positioned them somewhere outside of this picture, will have only a partial, will be able to retrieve a partial point cloud."}, {"start": 526.0, "end": 537.0, "text": " And I mean that's why it's a smart idea to train the NDF, this neural point descriptor in the same manner."}, {"start": 537.0, "end": 560.0, "text": " Okay, so why is this important? Why are we training this neural network to predict the occupancy function? So it turns out there are some nice properties that once we train the model the way I just described, so we are inputting partial point clouds and we're trying to basically output zero or one depending on whether this point x is inside of the shape or not."}, {"start": 560.0, "end": 567.0, "text": " And we do have the ground truth data because remember we're using synthetics, we can easily use synthetic data to train this model."}, {"start": 567.0, "end": 576.0, "text": " So now let's see what the interesting part and properties of this neural point descriptor field network FR."}, {"start": 576.0, "end": 592.0, "text": " So, this is here, to enable category level object manipulation, a spatial descriptor for a coordinate x given a point cloud P should encode information about the spatial relationship of x to the salient features of the object."}, {"start": 592.0, "end": 613.0, "text": " Our key insight is that the category level 3D reconstruction objective trains this network PHY, so that's the occupancy network, to be a hierarchical course defined feature extractor that encodes exactly this information, so the spatial relationship of x to the salient features of the object."}, {"start": 613.0, "end": 631.0, "text": " Basically PHY is a classifier whose decision boundaries is the surface of the object, so that's how it was trained. But like in a nutshell what they just said here is that they can use the representations that were learned during the training of this occupancy function PHY"}, {"start": 631.0, "end": 645.0, "text": " and use those as a way to capture these relationships of the point x to salient, geometrically salient parts of this shape."}, {"start": 645.0, "end": 658.0, "text": " Ideally what would happen is that these descriptors would be the same. So if I had a point x here, let me change the color here to something else, like if I had a point here next to this handle of this small mug"}, {"start": 658.0, "end": 672.0, "text": " and if I had a point here next to this handle of this tall mug, ideally the descriptors would be the same and that's what's going to enable the ultimate generalization that the authors showed."}, {"start": 672.0, "end": 681.0, "text": " So as I said their actual representation is going to be a concatenation of these representations here and they formulaically capture that exactly here."}, {"start": 681.0, "end": 693.0, "text": " So we thus propose to parametrize our neural point descriptor field F as the function that maps every 3D coordinate x to the vector of concatenated activations of PHY."}, {"start": 693.0, "end": 708.0, "text": " So basically, let me change the color, this is just a symbol for concatenation, we iterate through all of the layers from 1 through L, we grab the intermediate representations of the occupancy function of PHY"}, {"start": 708.0, "end": 721.0, "text": " and the notation is a bit sloppy here because we are not inputting point and the latent vector of the shape in every single of these layers, we are just inputting these in the first input layer."}, {"start": 721.0, "end": 725.0, "text": " It's a bit sloppy but you get the point."}, {"start": 725.0, "end": 734.0, "text": " Okay, so let me now explain, let me just show you one visualization, it's going to be a bit easier."}, {"start": 734.0, "end": 746.0, "text": " So as I said, ideally what would happen is that even if we had different mugs as you can see here and we have these green points, this is the x, ideally we would have the same descriptor."}, {"start": 746.0, "end": 749.0, "text": " So that's the whole point, that's the idea here."}, {"start": 749.0, "end": 756.0, "text": " And now there is a problem with this current formulation and the thing we just have done."}, {"start": 756.0, "end": 762.0, "text": " Basically, let me just kind of give you some understanding of this encoder here."}, {"start": 762.0, "end": 767.0, "text": " It's a point net from a paper that I think is from 2015 or something."}, {"start": 767.0, "end": 774.0, "text": " It's a very interesting architecture but the main idea of this architecture is the following."}, {"start": 774.0, "end": 782.0, "text": " The main idea is the following, you want to have this permutation invariant property."}, {"start": 782.0, "end": 790.0, "text": " So what the point net did is you input this array of points, so n times 3 because we have 3D points, we have n of those."}, {"start": 790.0, "end": 796.0, "text": " What we do is we use these shared MLPs and some transforms and we basically independently process each of these points."}, {"start": 796.0, "end": 803.0, "text": " So this point here is going to be independently processed from all of the other ones, at least in this layer."}, {"start": 803.0, "end": 810.0, "text": " Then we have the featuring transforms but the most important part is basically here and because we do a global pooling exactly here,"}, {"start": 810.0, "end": 817.0, "text": " that means that this final representation will be the same no matter how we permute this input here."}, {"start": 817.0, "end": 821.0, "text": " So that means that point net is permutation invariant."}, {"start": 821.0, "end": 825.0, "text": " But now imagine what happens if I were to translate."}, {"start": 825.0, "end": 830.0, "text": " So imagine we had a single point, imagine we had this shape like this mug here."}, {"start": 830.0, "end": 835.0, "text": " And imagine what would happen if I were to translate that mug right here."}, {"start": 835.0, "end": 842.0, "text": " What would happen is that all of the points in this point cloud would be basically modified such that each point would be,"}, {"start": 842.0, "end": 847.0, "text": " let's say, we had to add the translation vector t to those."}, {"start": 847.0, "end": 851.0, "text": " And now once you pass those here, you can imagine that the representation will change."}, {"start": 851.0, "end": 855.0, "text": " Okay, and the same thing would happen if I were to rotate the mug."}, {"start": 855.0, "end": 857.0, "text": " We have a different representation."}, {"start": 857.0, "end": 860.0, "text": " And that's all fine when we have a standalone point net."}, {"start": 860.0, "end": 867.0, "text": " But what I'm getting at here is that we want to make sure that if I were to rotate this mug here,"}, {"start": 867.0, "end": 872.0, "text": " let's say we have a point x here."}, {"start": 872.0, "end": 878.0, "text": " And if I were to rotate this mug, however we want, for the sake of simplicity,"}, {"start": 878.0, "end": 881.0, "text": " I'm just going to reduce this problem to a 2D problem."}, {"start": 881.0, "end": 890.0, "text": " So let's say we basically have a 2D square and we have a point x here."}, {"start": 890.0, "end": 899.0, "text": " Now if I were to rotate this square, so let's put some coordinate system here to make it a bit more clear."}, {"start": 899.0, "end": 903.0, "text": " Basically, if, now let's draw the coordinate system again."}, {"start": 903.0, "end": 912.0, "text": " And if we were to rotate the square for like 45 degrees or whatnot and rotate the point x the same way,"}, {"start": 912.0, "end": 918.0, "text": " so basically you'd want to ideally have the same descriptor for these two, right?"}, {"start": 918.0, "end": 922.0, "text": " Because their relative relationship has not changed."}, {"start": 922.0, "end": 928.0, "text": " And with the current implementation, remember if we were to change the 3D points,"}, {"start": 928.0, "end": 937.0, "text": " so if we were to change x and if we were to change the rotation or if we were to translate this object,"}, {"start": 937.0, "end": 941.0, "text": " as I said, this latent vector here is going to change, x is going to change,"}, {"start": 941.0, "end": 944.0, "text": " so that means that the descriptor is definitely going to change."}, {"start": 944.0, "end": 948.0, "text": " And in order to enable the generalization that they showed,"}, {"start": 948.0, "end": 960.0, "text": " it's very important to make this SE3 like an equivalent or just a fancy way of saying basically equivalent to translation and rotation."}, {"start": 960.0, "end": 966.0, "text": " And they also, I think they're kind of being sloppy with the equivalent versus invariance."}, {"start": 966.0, "end": 972.0, "text": " I've noticed that a couple of times, but yeah, hopefully you'll understand the gist of the paper."}, {"start": 972.0, "end": 980.0, "text": " So they say here that in other words, we require f to be invariant to joint transformation of x and p."}, {"start": 980.0, "end": 982.0, "text": " So again, f is, remember, the spatial descriptor,"}, {"start": 982.0, "end": 988.0, "text": " implying the descriptor field should be equivalent to SE3 transformation of p."}, {"start": 988.0, "end": 993.0, "text": " So here they just formulaically represent that if we were to transform both x and the point cloud p"}, {"start": 993.0, "end": 999.0, "text": " with some rotation and some translation, we want to make sure that the output of the function f,"}, {"start": 999.0, "end": 1003.0, "text": " remember, that's the descriptor, that's this whole thing is called f,"}, {"start": 1003.0, "end": 1007.0, "text": " and only the small portion here is called 5, that's the occupancy network."}, {"start": 1007.0, "end": 1011.0, "text": " But we care about this descriptor, the neural point descriptor field, okay?"}, {"start": 1011.0, "end": 1016.0, "text": " So as I said, we want to make sure that those outputs are the same."}, {"start": 1016.0, "end": 1024.0, "text": " They basically add two, let's call it tricks, to introduce this invariance or equivalence."}, {"start": 1024.0, "end": 1027.0, "text": " The first thing is fairly simple."}, {"start": 1027.0, "end": 1031.0, "text": " You basically first, if you want to be invariant to the translation,"}, {"start": 1031.0, "end": 1036.0, "text": " you can just find the mean point of the point cloud, so that's the mu here,"}, {"start": 1036.0, "end": 1043.0, "text": " and you subtract mu from both the input point x and from the point cloud p,"}, {"start": 1043.0, "end": 1049.0, "text": " and by doing this, you're guaranteed to always have this invariance."}, {"start": 1049.0, "end": 1055.0, "text": " To understand why that is, and hopefully it's very, very simple, but let me try and kind of make it a bit clearer."}, {"start": 1055.0, "end": 1059.0, "text": " If we have a coordinate system here, and I'm again going to be in a 2D space"}, {"start": 1059.0, "end": 1064.0, "text": " because it's harder to draw 3D objects using touchpad."}, {"start": 1064.0, "end": 1070.0, "text": " So if we had, let's say we have a square here and a point here,"}, {"start": 1070.0, "end": 1075.0, "text": " and let's say we have a translated version of this same square,"}, {"start": 1075.0, "end": 1081.0, "text": " so I'm just going to use a different color here, so let's say we have it here, okay?"}, {"start": 1081.0, "end": 1086.0, "text": " So if we were to find the mean point, which would be here for this object,"}, {"start": 1086.0, "end": 1090.0, "text": " and it would be basically here for this square, for the red square,"}, {"start": 1090.0, "end": 1095.0, "text": " and if we were to subtract those from the corresponding two cases we have,"}, {"start": 1095.0, "end": 1103.0, "text": " we'd end up with the same result both times, so we'd basically end up with this configuration,"}, {"start": 1103.0, "end": 1108.0, "text": " if we were to subtract this vector, and we would end up with the same configuration,"}, {"start": 1108.0, "end": 1111.0, "text": " if we were to basically subtract the other vector."}, {"start": 1111.0, "end": 1122.0, "text": " And because of that, the inputs are the same, and then obviously we do that even before we pass the data through the function f,"}, {"start": 1122.0, "end": 1126.0, "text": " and that's why we're going to be trivially have this property satisfied."}, {"start": 1126.0, "end": 1130.0, "text": " As for the rotation, it's not that simple."}, {"start": 1130.0, "end": 1136.0, "text": " You can do the same approach and kind of guarantee that you have invariance,"}, {"start": 1136.0, "end": 1142.0, "text": " because imagine if I were to kind of rotate this thing here,"}, {"start": 1142.0, "end": 1146.0, "text": " and now if we try to kind of just subtract the vector,"}, {"start": 1146.0, "end": 1150.0, "text": " we'd end up with a square that looks something like this,"}, {"start": 1150.0, "end": 1157.0, "text": " which obviously will be different input, and then f will just produce different output,"}, {"start": 1157.0, "end": 1159.0, "text": " so we don't have the invariance property."}, {"start": 1159.0, "end": 1162.0, "text": " So to achieve the rotation, active variance,"}, {"start": 1162.0, "end": 1167.0, "text": " they say here we rely on recently proposed vector neurons,"}, {"start": 1167.0, "end": 1173.0, "text": " which propose a network architecture that equips an occupancy network, i.e. the composition of epsilon,"}, {"start": 1173.0, "end": 1179.0, "text": " which is the point net encoder, and phi, which is basically the small submodule in the occupancy network,"}, {"start": 1179.0, "end": 1182.0, "text": " with full SO3 active variance."}, {"start": 1182.0, "end": 1188.0, "text": " So SO just means rotation invariance, active variance,"}, {"start": 1188.0, "end": 1193.0, "text": " whereas the SE3 means both translation and rotation, just keep that in mind."}, {"start": 1193.0, "end": 1198.0, "text": " And formulaically that means we want to achieve this, we want to achieve the same output,"}, {"start": 1198.0, "end": 1206.0, "text": " even if we were to transform the input point X and the point cloud P using some arbitrary rotation matrix R."}, {"start": 1206.0, "end": 1211.0, "text": " So how they achieve that is basically described in this paper called Vector Neurons."}, {"start": 1211.0, "end": 1215.0, "text": " I'm going to just give a super high level explanation here."}, {"start": 1215.0, "end": 1219.0, "text": " Basically what they've done is instead of using scalar neurons,"}, {"start": 1219.0, "end": 1222.0, "text": " so this is your classical linear layer in an MLP,"}, {"start": 1222.0, "end": 1226.0, "text": " they instead have as an output not a scalar but a vector,"}, {"start": 1226.0, "end": 1231.0, "text": " so that's a generalization of these scalar neurons."}, {"start": 1231.0, "end": 1236.0, "text": " And similarly, the implementation, this is, if you can guess,"}, {"start": 1236.0, "end": 1243.0, "text": " this is an implementation of a ReLU non-linearity in these vector neural networks,"}, {"start": 1243.0, "end": 1246.0, "text": " and basically you may be able to guess why this is,"}, {"start": 1246.0, "end": 1250.0, "text": " but in a scalar case you have something like this,"}, {"start": 1250.0, "end": 1255.0, "text": " so you have some input X scalar, you have output Y,"}, {"start": 1255.0, "end": 1261.0, "text": " and basically ReLU will kind of saturate this to zero, so this is zero,"}, {"start": 1261.0, "end": 1269.0, "text": " and then we're going to have for positive X, we're going to have a simple linear function, so Y equals X."}, {"start": 1269.0, "end": 1275.0, "text": " So here we have a similar, this is just a generalization of ReLU to 3D space."}, {"start": 1275.0, "end": 1278.0, "text": " Basically if we have vector Q, nothing will happen,"}, {"start": 1278.0, "end": 1285.0, "text": " so that's the same as if we were to have input, so let me just for clarity name this Q as well."}, {"start": 1285.0, "end": 1292.0, "text": " So in this case Q is a scalar, so in the case where Q is in the positive part of the domain here,"}, {"start": 1292.0, "end": 1294.0, "text": " Q remains unchanged."}, {"start": 1294.0, "end": 1301.0, "text": " Basically Q, because this is Y equals X, we're going to have the output will also be Q,"}, {"start": 1301.0, "end": 1303.0, "text": " and that's what we get here."}, {"start": 1303.0, "end": 1309.0, "text": " But in the other case where the Q lies in the, let's call it negative half space,"}, {"start": 1309.0, "end": 1312.0, "text": " which would be equivalent to this part here,"}, {"start": 1312.0, "end": 1319.0, "text": " we basically project the vector and we just keep the component that's alongside this plane,"}, {"start": 1319.0, "end": 1323.0, "text": " and we drop the component that's alongside negative K direction."}, {"start": 1323.0, "end": 1333.0, "text": " So yeah, it would take a whole video to explain why this works and why this introduces SO3 equivariance,"}, {"start": 1333.0, "end": 1339.0, "text": " but let's just take it as a theory and it just works."}, {"start": 1339.0, "end": 1348.0, "text": " So now we have all of the components necessary to understand how that robot is gripping those mugs,"}, {"start": 1348.0, "end": 1355.0, "text": " even if the pose and even if the shape, the actual instantiation of that mug is changing."}, {"start": 1355.0, "end": 1362.0, "text": " The idea lies in creating, inducing these energy landscapes,"}, {"start": 1362.0, "end": 1366.0, "text": " and we are using descriptors in the following manner."}, {"start": 1366.0, "end": 1371.0, "text": " Okay, let me use their visualization because it's going to be easier."}, {"start": 1371.0, "end": 1379.0, "text": " So imagine we have basically, this is P hat is, you can treat this as a demonstration,"}, {"start": 1379.0, "end": 1384.0, "text": " a mug that was used during demonstration, and P is the one used during the test time."}, {"start": 1384.0, "end": 1391.0, "text": " So imagine we have this point X here in green and we have the shape here,"}, {"start": 1391.0, "end": 1397.0, "text": " and basically we can pass those and we get out, as the output we get the description,"}, {"start": 1397.0, "end": 1403.0, "text": " the descriptor of their mutual relationship. Let's put it that way."}, {"start": 1403.0, "end": 1408.0, "text": " On the other hand, we have novel point cloud P here."}, {"start": 1408.0, "end": 1413.0, "text": " It's, as you can see, a bit differently shaped. It's not the same as the one above."}, {"start": 1413.0, "end": 1420.0, "text": " And now the idea is to find corresponding,"}, {"start": 1420.0, "end": 1428.0, "text": " what will be the corresponding point to this X for this current configuration here?"}, {"start": 1428.0, "end": 1432.0, "text": " And the answer you would give intuitively is it should be somewhere here, right?"}, {"start": 1432.0, "end": 1436.0, "text": " And the reason it's going to work is because, as you can remember,"}, {"start": 1436.0, "end": 1441.0, "text": " how we train the F function, or neural descriptor,"}, {"start": 1441.0, "end": 1446.0, "text": " is such that this is what the energy landscape is going to look like."}, {"start": 1446.0, "end": 1450.0, "text": " So basically for points which are close to the handle here,"}, {"start": 1450.0, "end": 1455.0, "text": " because remember the shape is going to be abstracted away because of the point net"}, {"start": 1455.0, "end": 1460.0, "text": " and the way we trained the whole system,"}, {"start": 1460.0, "end": 1464.0, "text": " and that's going to cause the energy landscape to look like this,"}, {"start": 1464.0, "end": 1468.0, "text": " which means once we use simple optimization, so we'll be tweaking,"}, {"start": 1468.0, "end": 1473.0, "text": " so what will happen is we'll be using Adam, so they've been using, in practice they've been using Adam,"}, {"start": 1473.0, "end": 1478.0, "text": " so we'll be tweaking the parameters of X, so basically three scalars,"}, {"start": 1478.0, "end": 1485.0, "text": " we're just tweaking those such that point X will be moving towards this low energy part of the landscape,"}, {"start": 1485.0, "end": 1491.0, "text": " and once that happens, basically this, so yeah, this is this loss,"}, {"start": 1491.0, "end": 1494.0, "text": " we are minimizing that, and as we are minimizing that,"}, {"start": 1494.0, "end": 1500.0, "text": " we're basically finding this point that we just found intuitively a couple moments ago."}, {"start": 1500.0, "end": 1505.0, "text": " And because of that, the robot will basically know how to grip this handle."}, {"start": 1505.0, "end": 1509.0, "text": " There will be like a simple high level explanation of how the thing works."}, {"start": 1509.0, "end": 1517.0, "text": " So now the problem is with this thing is that the grip of a robot actually is not a single point,"}, {"start": 1517.0, "end": 1523.0, "text": " because if you have a single point, you can have multiple frames, coordinate frames attached,"}, {"start": 1523.0, "end": 1526.0, "text": " you can have infinite coordinate frames attached to that point,"}, {"start": 1526.0, "end": 1534.0, "text": " so they just kind of generalize this to the case where we want to have the pose information as well,"}, {"start": 1534.0, "end": 1542.0, "text": " and how they represent the pose is very similar, you just basically concatenate,"}, {"start": 1542.0, "end": 1546.0, "text": " so you take all of the points here, so this is again concatenation symbol,"}, {"start": 1546.0, "end": 1548.0, "text": " you take all of the points from this query point cloud,"}, {"start": 1548.0, "end": 1552.0, "text": " so those are these points with various colors as you can see here,"}, {"start": 1552.0, "end": 1556.0, "text": " and you just basically concatenate their descriptions,"}, {"start": 1556.0, "end": 1561.0, "text": " and that's how you get the final description for the pose."}, {"start": 1561.0, "end": 1566.0, "text": " Once you have that, by the way, in order to get those points,"}, {"start": 1566.0, "end": 1572.0, "text": " they mentioned here that you're using a robust heuristic to sample points uniformly at random"}, {"start": 1572.0, "end": 1576.0, "text": " from within the bounding box of the rigidbody S,"}, {"start": 1576.0, "end": 1581.0, "text": " whereas you can think of S as like a gripper in the case of picking up an object,"}, {"start": 1581.0, "end": 1586.0, "text": " and in the case when you want to place it, it's going to be some other object,"}, {"start": 1586.0, "end": 1596.0, "text": " and in the case when you want to place the actual shape onto some table or whatnot, onto a shelf,"}, {"start": 1596.0, "end": 1603.0, "text": " you won't have a gripper, you have some different salient object,"}, {"start": 1603.0, "end": 1611.0, "text": " and basically what they do is, as you can see here, finally they basically do K demonstrations,"}, {"start": 1611.0, "end": 1615.0, "text": " so 1 through K, and they collect these descriptors,"}, {"start": 1615.0, "end": 1620.0, "text": " so for where the gripper was during picking and where the placement surface was"}, {"start": 1620.0, "end": 1622.0, "text": " when the object was placed in the demonstration,"}, {"start": 1622.0, "end": 1627.0, "text": " and then they just average out all of these descriptors,"}, {"start": 1627.0, "end": 1636.0, "text": " and they end up with a tuple, basically this Z pick and Z rel with these bars on top of them,"}, {"start": 1636.0, "end": 1639.0, "text": " that's a kind of silly notation, but yeah."}, {"start": 1639.0, "end": 1645.0, "text": " And as I said, once you have the descriptors, you can just minimize this energy landscape"}, {"start": 1645.0, "end": 1655.0, "text": " to get the position for the novel object in a novel instance of the same class"}, {"start": 1655.0, "end": 1659.0, "text": " with a different orientation or whatnot."}, {"start": 1659.0, "end": 1665.0, "text": " And again, they mention here, we rely on off-the-shelf inverse kinematics and motion planning algorithms"}, {"start": 1665.0, "end": 1668.0, "text": " to execute the final predicted pick and place task,"}, {"start": 1668.0, "end": 1673.0, "text": " so that means that the whole trajectory between the two endpoints"}, {"start": 1673.0, "end": 1678.0, "text": " is not something they were trying to solve in this paper."}, {"start": 1678.0, "end": 1683.0, "text": " Yeah, here's just some visualizations of how you start up with some random points,"}, {"start": 1683.0, "end": 1687.0, "text": " and then because the demonstration was such,"}, {"start": 1687.0, "end": 1693.0, "text": " and then that means we had certain descriptor as an output of this demonstration,"}, {"start": 1693.0, "end": 1701.0, "text": " now we just minimize the loss, so we move towards the lower parts of the energy landscape,"}, {"start": 1701.0, "end": 1709.0, "text": " and we end up with the same, as you can see, analogous position for this novel shape,"}, {"start": 1709.0, "end": 1714.0, "text": " so basically this test shape is potentially different than the demonstration one."}, {"start": 1714.0, "end": 1719.0, "text": " Results are, surprise, surprise, way better compared to some of the baselines,"}, {"start": 1719.0, "end": 1725.0, "text": " especially when you're using upright pose, the baseline is kind of on-pair,"}, {"start": 1725.0, "end": 1729.0, "text": " let's call it on-pair, but its NDF is still better compared to DON,"}, {"start": 1729.0, "end": 1734.0, "text": " but like when you start using arbitrary poses, that's when this thing starts shining,"}, {"start": 1734.0, "end": 1739.0, "text": " and it's way better compared, as you can see the gap increases significantly here and here."}, {"start": 1739.0, "end": 1744.0, "text": " Also, you can notice that there is some drop in the performance here,"}, {"start": 1744.0, "end": 1750.0, "text": " and I think that they mentioned that the main reason was that the point clouds kind of changed,"}, {"start": 1750.0, "end": 1755.0, "text": " because during the demonstrations the robots only saw the mug or whatever object they were using"}, {"start": 1755.0, "end": 1762.0, "text": " from the bird view perspective, and when they use arbitrary pose,"}, {"start": 1762.0, "end": 1766.0, "text": " obviously there are some novel occlusions and these occlusions that happen,"}, {"start": 1766.0, "end": 1769.0, "text": " and so that's why there is some degradation there."}, {"start": 1769.0, "end": 1776.0, "text": " That's pretty much it. Here they just show some ablations, basically it's better to use,"}, {"start": 1776.0, "end": 1785.0, "text": " like should we use final layer or the first layer of the occupancy network or all layers,"}, {"start": 1785.0, "end": 1790.0, "text": " and we saw that they ended up using all of these layers as a representation,"}, {"start": 1790.0, "end": 1797.0, "text": " and that turned out to be the best option, and yeah, that's pretty much it."}, {"start": 1797.0, "end": 1801.0, "text": " There is a couple of things they also mentioned here."}, {"start": 1801.0, "end": 1806.0, "text": " Yeah, they also showed that as the number of demonstrations increases,"}, {"start": 1806.0, "end": 1811.0, "text": " you can see that the performance is increasing, so it's important that they use around 10 demonstrations"}, {"start": 1811.0, "end": 1816.0, "text": " so that this can work, so this is a few-shot learning, let's say,"}, {"start": 1816.0, "end": 1823.0, "text": " and as a point of further potential research, they say that further NDFs can only define"}, {"start": 1823.0, "end": 1826.0, "text": " transferable energy landscapes over poses and points."}, {"start": 1826.0, "end": 1831.0, "text": " Future work may explore integrating such energy functions with trajectory optimization"}, {"start": 1831.0, "end": 1833.0, "text": " to enable NDFs to transfer to full trajectory."}, {"start": 1833.0, "end": 1839.0, "text": " So that's the part I said that the actual movement between the picking up and placing"}, {"start": 1839.0, "end": 1843.0, "text": " is done using these off-the-shelf methods."}, {"start": 1843.0, "end": 1850.0, "text": " So I love the idea. There are things that, in my opinion, should be improved upon, obviously,"}, {"start": 1850.0, "end": 1857.0, "text": " not using the heuristics, not using this off-the-shelf algorithms to plan the trajectory,"}, {"start": 1857.0, "end": 1863.0, "text": " but maybe inducing an energy loss scape across this whole thing."}, {"start": 1863.0, "end": 1866.0, "text": " Also, I'm always trying to think when I see a method like this,"}, {"start": 1866.0, "end": 1868.0, "text": " whether our brain is doing something similar."}, {"start": 1868.0, "end": 1873.0, "text": " It's not that we have to copycat our brain, but we need to find some inspiration there,"}, {"start": 1873.0, "end": 1879.0, "text": " and it's kind of the only current implementation of AGI that we have present in the world,"}, {"start": 1879.0, "end": 1881.0, "text": " so I guess it's a good starting point."}, {"start": 1881.0, "end": 1891.0, "text": " But just kind of doing gradient descent on the energy landscape doesn't seem like the thing"}, {"start": 1891.0, "end": 1892.0, "text": " that our brain is doing."}, {"start": 1892.0, "end": 1900.0, "text": " But when I saw the demos they showed, it's quite impressive, and basically, yeah,"}, {"start": 1900.0, "end": 1902.0, "text": " kudos to the authors."}, {"start": 1902.0, "end": 1906.0, "text": " Some parts of the paper could have been written a little bit more clear."}, {"start": 1906.0, "end": 1913.0, "text": " Yeah, at times it's a bit hard to understand, and the 10-pager format for the papers is not helping"}, {"start": 1913.0, "end": 1919.0, "text": " that much because people have to squeeze in stuff, and then you lose the structuring of the documents."}, {"start": 1919.0, "end": 1922.0, "text": " It's hard to read, and yeah."}, {"start": 1922.0, "end": 1924.0, "text": " That said, hope you like this video."}, {"start": 1924.0, "end": 1930.0, "text": " As I said, it's a bit different because I still don't have my setup here in London,"}, {"start": 1930.0, "end": 1934.0, "text": " but I'll be continuing with the Jack series."}, {"start": 1934.0, "end": 1942.0, "text": " Next up, you can expect my Flex video, Haiku video, and then I'll start doing some videos on ML Ops,"}, {"start": 1942.0, "end": 1944.0, "text": " and yeah, stay tuned for that."}, {"start": 1944.0, "end": 1957.0, "text": " Until next time, bye-bye."}]