Non-Parametric Transformers | Paper explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I do a deep dive of the "Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning" paper.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Non-Parametric Transformers paper: https://arxiv.org/abs/2106.02584
✅ Jay Alammar's BERT blog: http://jalammar.github.io/illustrated-bert/
✅ My LinkedIn post (Judea Pearl): https://www.linkedin.com/posts/aleksagordic_pearl-causality-intelligence-activity-6807985607432785920-Zxn2 (also check out my other posts I made related to this)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Key ideas of the paper
01:40 Abstract
02:55 Note on k-NN (non-parametric machine learning)
04:30 Data and NPT setup explained
06:15 NPT loss is inspired by BERT
08:20 A high-level architecture overview
11:30 NPT jointly learns imputation and prediction
12:50 Architecture deep dive (input embeddings, etc)
20:45 More details on the stochastic masking loss
23:30 Connections to Graph Neural Networks and CNNs
29:45 NPT achieves great results on tabular data benchmarks
34:25 NPT learns the underlying relational, causal mechanisms
39:40 NPT does rely on other datapoints
42:10 NPT attends to similar vectors 
45:15 Conclusions

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Petar Veličković
Zvonimir Sabljic
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#nonparametric #transformers #relationalreasoning
 What's up? In this video I'm covering this new exciting paper called Self-Attention Between Data Points Going Beyond Individual Input-Output Pairs in Deep Learning, which introduces this model called non-parametric transformers. And now, non-parametric doesn't mean it doesn't have parameters, it's quite the opposite. So it just means that it doesn't scale well with data. Basically, the more data you give it, the more parameters it's going to need. And in general, these non-parametric models either have this time-wise or space-wise slash memory-wise dependency on the number of data points. So having said that, the main idea this paper introduces is that it's a kind of attempt to generalize even transformers. So whereas transformers basically assume there is a fully connected graph between the input tokens, and so they kind of tend to everything, here they additionally actually attend over different data points. So that means it's assuming an implicit fully connected graph over the data points. And I'll get into those details in a moment. And basically, even during the inference, they need the training data points, and they'll use those to better infer the class or the value if you're doing regression of the test data point. And we'll see all of those details. So that's like the main innovation that this paper brings. They get a really nice results on some tabular data sets, which is something that previous methods like XGBoost were dominating. And yeah, that's pretty exciting. So having said all that, let me jump into the details of the paper. They say here, we challenge a common assumption, underlying most supervised deep learning, that our model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general purpose deep learning architecture that takes as input the entire data set, instead of processing one data point at a time. Our approach uses self attention to reason about relationships between data points explicitly, which can be seen as realizing non-parametric models using parametric attention mechanism. And again, you can see here, so entire data set, they are learning from the entire data set, and they're doing inference, as we'll see soon see on the pretty much, because it doesn't scale well with data, they'll have some simplifying assumptions and approximations. So they'll be using some chunk of the training data to do inference on. As I said, the model is called non-parametric transformers. And so a quick note here, like just on like conventional non-parametric models. So they say here, conventional non-parametric models cannot learn in the sense familiar to deep learning practitioners. Interactions from the data limiting the flexibility these models have in adapting to the data at hand. And the good example would be a KNN, K nearest neighbors algorithm. So just a quick note how it works, and you'll then understand what non-parametric models mean. So let's say we have some classes here, like maybe we have two classes, one is like triangles, and then maybe we have something like circles, something like this. So and assume, so this is your data set, and now assume I give you a query point and ask you what's the class of this data point. And now if we assume that K equals three, basically that means we'll find the three nearest neighbors, hence the K nearest neighbors algorithm. So that's going to be this, and then we'll output by majority voting, let's take that simple variant of the algorithm without including the actual distances, and either way we're going to have triangle here. So as you can see, the more data you have here, so if we had a bunch of different circles and triangles, and if K goes to infinity, and also assuming data goes to infinity, that means that this algorithm doesn't scale well, because you'll have to search for like infinite number of nearest neighbors. And so yeah, contrast that to maybe a neural network where no matter the like the number of data points, the size of your data set, you'll have a specified pre-specified architecture whose parameters won't change, and the inference time won't change, because you always have to pass a single data point or a batch whatever through the neural network. So that's the difference between parametric and non-parametric, and as we'll soon see, now we'll see how the model actually works. So I first want to explain you the architecture itself and the data layout. So you can see here, their data set is tabular, so they basically assume they concentrate on like on this single target regression and classification. That means the following. That means they are placing the attributes along, so these are data points, as you can see, along this axis. We have here attributes, but in this last column, they're going to assume they're having like targets, either classification targets, like what's the the class for these. So for this data point here, we have attributes, and this is going to be some some like class, okay? Or if we're doing regression, there's going to be some continuous value. So that's just how they structure it, and as you can see here, if we had a parametric model, maybe a transformer, this is how the inference would look like, except for the transformer, you'd have edges going here as well, and here as well, because we are attending, every token is attending to every token, i.e. every attribute is attending to every attribute. So finally, we output the the target value, whatever that is, either class or a continuous value. And here in MPT, as you can see, they're also leveraging other data points, so maybe some training data points, and basically using all that information, they can reason and figure out this value, and they show that this inductive bias is really useful, so having the ability to attend to other data points as well. So that's an interesting shifting perspective, because usually we're used to this setup, but yeah, that's cool. Okay, so let me start explaining how the thing works, and first things first, like the objective is the following. So MPTs, so non-parametric transformers, training objective is to reconstruct a corruptive version of the input data set. Similar to BERT, we apply stochastic masking to both features and targets, and minimize a loss on MPTs prediction at entries masked out in the input. So basically, they'll have a binary mask telling them which parts are masked here, and those will be predicted. And a quick side note, if you're not familiar with BERT, it's very simple, it's a famous transformer, and the way it works, the way they train BERT is the following. So you have input tokens, so this is some sentence, like help print Mayuko, and by the way, this is from JLMR's blog, I'll link it down in the description, and what they do when they train BERT is they mask out certain tokens. So this Mayuko will be masked by like a special symbol called mask, whatever, with some probability, or they'll just place some random value there, okay? And they do that for a number of tokens, and now the goal is to output Mayuko here, so output the correct class up here, so if they masked some token here, they'll have to output it here. So the final loss will just be, if you're doing classification, like they'll just do like basically cross entropy loss here, and they're gonna back prop that and train the model. So it's a self-supervised way of training, and I'm just ignoring, for the time being, for those of you who know how BERT works, I'm just ignoring this part of next sentence prediction, and focusing on these, because that's the part that's really similar in MPTs, they're doing the same thing as BERT, pretty much. Okay, so they also assume, as I said, that the targets are the final attribute of this table. So as I said, they're gonna mask out these values, and they're gonna try predict those values, but they're also going to mask out some of the attributes as well, so you'll see, that's a combination. Let's first focus on the high-level picture. So this is, as I said, this is the input, so this is your tabular data set, you have n data points, you have d attributes, and some of the values are masked, so either attributes or the target values. So what I do is, they first embed this data into this n, d, and now we have dimension e here, and I'll get into a bit more details of how exactly the input and output embeddings work, but having done that, somehow, consider it a black box for now, what I do is the following. So they apply, so they first flatten out this thing, and they apply a data point, so basically the same thing as what was finally did in the original transformer, so we have self-attention from the original transformer applied here, across data points first. So they basically attend, as you can see these lines here, every data point attends to every other data point, so we do a VASVANI type self-attention, then they reshape it, and then they do the same thing just across the, this time across attributes, and they repeat this n times, basically as a for loop, and then they didn't draw it here, but basically they have, again, they have output linear projection layer, and that's the whole thing, the architecture is pretty simple. Now if you're familiar with recently published MLP mixer, and I'll link it somewhere here if you haven't watched my video, I've explained it there, but like, you can see that if we treat patch like a data point, so MLP mixer like inputs an image, you split it into patches, and if we treat those patches as data points, there is a similarity in the mixing part at least, so they also have, so basically they also have the first part where they are attending over different channels, over different patches, so different data points, but the same channel, and here they have the part where they are attending basically the samehow where they are attending basically the same patch, but different channels, so that will be, that will correspond to different attributes here. So this corresponds to this part, and this part where they are attending over different data points corresponds to this part, where they are attending over different patches. they're attending over different patches. So I just thought that's a neat analogy and other than that they are not using attendance. So the whole point of MLP mixture paper was to ditch the attention and just focus on the MLPs so I have just have less even less inductive biases in this architecture. Okay there was a small digression hopefully you found that connection useful. Now let me go and try and focus on explaining the the actual architecture. So quick mention again we focus on single target classification and regression corresponding to a masking matrix M with ones at all entries of the table column X that this is just a means that the last column is masked but online multi-target settings imputation self-supervision using input feature is also possible. So that's what I told you when I said like basically they can also mask some other parts of the of the input data and that's really cool. So the cool part here is that and I noted it here basically it joins the learns imputation plus prediction. So in conventional approaches what you usually have to do when you use those boosting methods you have to have a two-stage approach. The first stage just imputes the values and now again imputation just means that you're missing some values. If this is your tabular data and let's assume you're missing some value and I'll just denote that as a question mark these algorithms usually have to maybe apply some heuristic like maybe they'll have to attend over the same row and maybe find I don't know like mean or median and impute it right here or they can even train a dedicated machine learning model which will learn how to impute those values. Once the imputation is done that stage is done now they only can do prediction. Here this model MPT does this jointly and that's really cool because it can learn much better how to do imputation in this end-to-end deep learning scenario because basically it can also attend other data points and make much better much more educated decisions here. So that's something I thought mentioning and yeah I'll get back to this equivariance thing and I'll make some nice connections to graph neural networks but like for the time being let me just again I showed you the the high-level picture of how the architecture works now let me get into a bit more details. So again I won't be explaining the actual self-attention mechanism I have a video I made a video on that you can check it out it's a basically Vasvani self-attention so you've been seeing if you're following your like recent research this is being used everywhere so I'll skip that part and I'll just focus on explaining the input output embeddings. So let me just scroll down in the appendix okay it's gonna be so okay by the way it's the paper is really neat they have they added really cool things like checklists and computational requirements and also some implicated like societal implications so I really love how the paper is written and yeah kudos to the authors. Okay let's see how the how the thing works in the initial stage of the pipeline of the architecture so basically we have we have a data set so this is the data set tabular so we have and data points we have the attributes one of them is also the the target column so implicitly that's the last column and how it works is the following so they say it here we'll learn separate embedding weights for each attribute and that makes sense because each attribute contains its own semantics and we'll now see like even like in more details why that is so let's say we have this column so let's assume this this column contains categorical variables so we'll have a certain number of classes there and the first step they do for a categorical variable is they just basically encoded into a one-hot encoding so this is going to be size of C where C is the number of classes and basically so so let's say if we if you have a number I don't know two and assume we have I don't know maybe four classes this is going to be embedded into a one-hot vector like this so basically we have four slots here and one goes here zeros elsewhere okay so they're gonna embed every single attribute in that column across different data points into that one-hot embedding okay so the next step they do is they append the the mask here so they'll append the mask down all the way down right and they do this for every single column so now let's assume this one is maybe continuous variable the first thing they do here is they standardize the column so they just make it have mean of zero and standard standard deviation of one or variance equal to one and basically now after they do that they'll just have a single dimension here so that's one not C and they'll append the mask again okay so that's gonna be two and as you can see and also some different column may have different number of classes here so it may be C C prime whatever and that means for that reason we'll basically have to have different independent linear projection layers not just because of the semantics but also because of the pure shape mismatch so you can think of it like both ways so once we have that we just have like linear projection layer so which will just project all of these so whatever the number of dimensions here is into e dimensions so this is going to be kind of projected into like you can imagine having a fully connected layer here blah blah I don't know how to draw today but basically you can imagine it like this is you'll have e dimensions and we'll end up with a n times d times e volume the same thing you saw on the high level diagram so that's the embedding part almost let me just show you the formula so this in code this is the part that embeds the categorical variables into one hot embeddings then they append the mask so again the mask will just have dimensions n times d and if you have one somewhere and I forgot to mention that part if you have one somewhere that means that part is going to be masked out so let's assume this part here has one so that means they do the following aside from appending one here at the last like slot here they're going to mask out all of this with high probability so all of these will get set to zero and so that means that that one hot vector will just be zeroed out so this hot this one hot vector will just be zeroed out so we'll have just all zeros all the way here so as in birth this is either going to be zeroed out or they're gonna just set our random value in there and that's how the thing works so again getting back to to the formula you can see after we do the one hot encoding for categorical variables or just normalization for continuous variables we append the mask we project this into the into this e dimensions and then we just concatenate across all the different columns IE attributes and then they additionally have these learned embeddings for index and type I'm not sure whether they did ablation on whether they actually need the both of these whether it can be jointly learned like the type and index but yeah if anybody knows or the authors are watching the video they can maybe explain it all in the comments having done that for a single attribute column so this is for a single attribute column so this was this part maybe we're focusing only on this specific column on this volume now they just concatenate all of those as you can see across the axis D so this is the D axis and that's it that's it you get the NDE volume and similarly for the output embedding I'll just briefly explain it so here they're not predicting the mass variable so they'll just end up with so EJ was the notation they used to denote this part like without the the masking dimension that's added so they're just going to again linearly project all of these this E into into E let's call it subscript subscript D where this is basically depends on the on the column so for this column this ED is going to be equal to C for this continuous variable column is going to be equal to 1 etc so yeah that's basically needed so that we can then do the like appropriate apply the appropriate loss across that column so it's very easy they say here so it's one for continuous attributes it's the number of categories for a categorical attribute and to obtain the final prediction matrix we take the argmax over the categorical predictions so that's how they do the prediction so basically whatever they spit out in the last layer maybe like they'll have some probabilities obviously if we extract this part here and then just make it a bit nicer to see so if I extract it here it's going to be some distribution and like dimensions here are going to be C as I said so it's gonna be just some distribution of across the the classes and they're just gonna do like like normal like argmax and find the the the highest probability class and that's their prediction okay that was lots of details on the on the architecture let me let me get back to the paper and tell you some interesting stuff so first things first I don't want to finish with that with the loss so as I said before so they're applying stochastic feature masking they're applying stochastic target masking similarly to BERT and during the training we compute the negative log likelihood loss at training targets as well as the auxiliary loss from masked out features so that's the part I mentioned so assuming we have a classification problem what I do is the following so they'll output again this would be the output of their model of the NPT and let's focus on a single attribute on a single slot here basically they said this is going to be C and they're simply going to take the the ground truth position so maybe this is the ground truth class they're gonna take the probability that they had there so let me just kind of draw it three-dimensional here so they're just going to extract the probability there and they're just gonna do minus log p of I so in order to minimize the loss you want to have your obviously so that's like cross entropy simple stuff you want to have your probability going to one this is the output PI and this is the loss okay the y-axis so they just summed that out across all of the masked out features so assume we had one here for for mask assume we had maybe if we had one here we're gonna additionally extract the class for that one and do minus log PI and we're gonna just do a sum over those and just do a back prop through the model hopefully that was clear so long sure store they're just going to sum up across all of the masked by their features or targets here and they're going to do a sum and just do a big prop on their loss okay so interestingly stochastic target masking means that many training targets are unmasked to the model at training time so this is interesting that means that this column which contains the targets will not be like completely massive some parts will have zeros here and that means that the model can actually see that the correct like target value for that for that data point this allows MPTs to learn to predict at each epoch the masked targets of certain training data points using the targets of other training data points in addition to all training data features okay so that's an interesting part and we'll see how that helps them do relational reasoning later on here's just a short note I'll obviously this approach will have some problems with scaling because they are attending like bunch of training data points during inference as well and yeah that's something that future research will need to tackle with okay architecture explained now let me actually show you some exciting results and first give you some connections to graph neural networks that I promised here so first things first they sit here MPTs are equivalent to a permutation of the data points in other words if the set of input data points are shuffled so if we have the tabular data you have and and and rows d columns so we kind of shuffle these rows MPTs produce the same predictions but shuffled in an analogous manner so that's the equivalent part so it should not depend on their ordering that's the important part so now let me just kind of give you some connections with CNN's and graph neural networks and hopefully you'll find that useful okay where are we let me just find the yep here it is so they mentioned here that MPTs can be seen as a conceptual generalization of graph convolutional networks and again I've covered that in my previous video you can check it out I'll link it somewhere here and the other graph neural networks like get again I've checked I've covered that one as well so that's graph attention network engine in which a set of dependencies edges between data points is not known a priori and instead is learned from data using self-attention so this is the interesting part and now let me let me let me show you what they mean by that so amputees learn an implicit graph structure so whereas if you have if you're familiar with graph neural networks let me let me draw a graph here basically you have some notes in your graph and for example for forget you'll have explicit connections here okay you'll have something like this and then what graph attention network will do it will just kind of assign certain attention coefficients to each of these edges and it's going to aggregate all of these node features so every node has a feature vector associated with it it's going to kind of aggregate all of these by some like invariant like mapping and they're going to get the final embedding for this vector so this was maybe so prior to this we had this vector had like embedding vector V and now after this accumulation across the neighborhood so we're somehow modifying these feature vectors we are somehow combining them and we get V prime that's the updated representation so now the difference here is between get and in general the graph neural networks is they have explicit graph structure so compared to that amputees the nonparametric transformers now do the following it is something similar that transformer there with like like with with sentences basically it assumes a fully connected graph so it assumes that all of these are connected with each other okay I think I've connected everything basically and now it just learns which of these edges have what kind of strength and so it's implicitly inferring the structure of the graph without being said like which edges exist on which edges does not exist so it's a more of a soft assignment here compared to graph neural network so that's the first connection second things second thing I want to mention is that like graph attention network is invariant to the data point ordering so it's not active variant it's invariant that means no matter how you have these node feature vectors in your table so basically let's say this is one this is two this is three you'll represent your graph as certain table where again here we have data points and here you have number of attributes so now no matter if you if you shuffle these one two three like in whatever permutation you'll still have the same V prime feature vector here because what graph attention network does is it just sums sums up these these these values and basically some enables us to have these this invariance property right because some doesn't care about the permutation of your input so on the other hand let me take a convolutional neural network just to explain the active variance part I think you'll find it useful basically let's assume we have an image here and in this image we have certain like an image of a square okay and now if we pass that to a CNN and let's assume we're in the first layer of the convolutional neural network and that the the first kernel the first filter learns to just figure out the vertical edges in in your in your image so what the image will look like is something like this we'll have something like this and we'll have like two vertical edges here okay so the active variance means the following if we now take this square and we translate it here maybe here now what will happen in the output is this response will just shift by the same amount here so as you can see it's equivalent it's varying with input it's not invariant to the position of the square it's actually varying if that's even a verb with the input okay so the invariant part income nets usually comes at the like at the classification head part because there you really don't care you just care that you have a square in the image if you're trying to classify it you don't care where it is so it's actually desirable property to have invariance at the latest stages of the CNNs but like first stages have active variance so that was just a short summary of what's a covariance what's invariance and so basically now that we know that we can say two things first things first is that amputees are active variant second thing is that they are implicitly learning the graph structure over the data set and that's really cool but unfortunately it's a kind of computationally expensive and yeah they say here we'll leave further exploration of the close connection between amputees and graph neural networks to future work I just briefly mentioned a couple of obvious properties I guess some of the more knowledgeable people from the graph ML community could write down comments of what interesting other properties this thing has with graph neural networks okay that was it now let me jump to experiments they have some really nice experiments okay the first experiment they do is they they basically test this on a tabular data set a collection of tabular data sets ten of them to be precise and they say it here amputees perform competitively on established benchmarks tabular data is ubiquitous in real-world machine learning but notoriously challenging for general purpose deep neural networks which consistently underperform boosting methods and are rarely used in practice so again short note boosting methods they are super simple you basically just take take a weak learner which means it's a some some really weak model that can be a bit better than random guessing and they combine these in a clever way for example majority vote or something to get something called strong learner which actually has some nice accuracy let's say if we're doing classification so now there was a bunch of like different work that came after the original boosting method so there is first thing is this add a boost so there's just adaptive boosting where once you create the first week a classifier let's say you basically use the data points is performing poor on to feed those into the next week learner which will learn to perform much better on the points where this first learner was having difficulties with so that's the add a boost and we had gradient boosting and then there is a bunch of variations on gradient boosting so here you can see XGBoost which is just a performant like implementation of that gradient boosting method we have cat boost also gradient boosting implementation light gradient boosting method again gradient boosting and yeah so you get the point these boosting methods are pretty popular on tabular data you'll see them all around kegel and now let's see the results they got and they're pretty spectacular if you take a look at this binary classification tasks so I think there was four of these you can see MPT is the first in this table it's better than then cat boost and light GBM actually boosts altogether it's also the first on this table where they're doing multi-class classification and MPT is again the best method it's got a bit less variance compared to XGBoost they're otherwise tied here and finally we have regression tasks where MPT was a bit worse so you can see it's tied with XGBoost it has a little bit bigger variance and cat boost is kind of really really good for this for this task but nonetheless they mentioned it here so in addition to its strong rank-wise performance MPT achieves best performance on four out of ten benchmark datasets more than any other method so they definitely proven that this neural network can be used for tabular data and give some nice results by the way if you're asking yourself maybe it's not fair like they're they're probably used much more computation to train the MPT that's actually not true because for just figuring out the best hyperparameters for those boosting methods they actually wrote something some somewhere in the paper that the computation necessary was pretty much on pair so as so this is supporting our hypothesis that attention between data points is a useful architectural inductive bias for prediction and aside from these tabular datasets they also tested MPT on the image kind of benchmarks like I think MNIST and Cypher-10 and they said here similar to previous work on transformers for computer vision we would expect pre-training a millions of images to significantly boost MPT's performance and if you watch my previous video you now know that vision transformers and MLP mixers don't need pre-training at all if you use this thing called sharpness aware minimization objective and again I'll link the video somewhere here you can check it out basically if we have if you somehow find a way to to smoothen out the lost landscape you'll notice that you don't need extensive pre-training anymore and you can achieve results that outperform resonant baselines and that's it in a nutshell so I guess the authors of the paper either either didn't have the time to rewrite this or they still haven't read that paper which recently came out a couple of days ago okay that's the first part now let's see some some some cool things this diagram here let me just introduce it basically they make this semi synthetic the problem and they say it here for each batch we input the original data with masked target values as well as a copy of the original data where all target values have been revealed I know masking is applied at this time we input novel semi synthetic test data to ensure that MPT has learned the correct relational mechanism as opposed to just memorizing the target values okay so they they have a nice nice chart here where you can see the what's what's the input so this is the input we have some data points and we have the target values there are hidden denoted by these question marks and we have the actual correct values here so if MPT smart even though we're gonna just back prop on these masked values is going to learn how to attend how to find this similar vector so this is basically the same vector and then just copy this value here instead of learning to somehow map and use the attributes is going to learn the relational mechanism and why do we want that well because it's going to generalize much better if it learns how to just attend to the same vector and just copy the value that's the whole point of relational reasoning so as you can see here they draw some attention maps so this green data point so that's this one the first one it attends to this one so that's this data point here so that means it learned completely and you can see attention here is one all around you have zeros if you take a look at this heat map so that means the whole attention of this data point of the first data point is on this one and that's exactly what we wanted and you can see the pattern going for all of these data points here and here this part you can ignore they didn't enforce this they did write something about this in the appendix why these data points are attending to themselves so that's something may be interesting taking a look into but yeah basically you can see the results are really cool Pearson correlation is 99.9 that means basically a fancy way of saying that it basically learns how to copy over these values or that's a hypothesis for now but like it's it's it's basically outputting perfect values as if it was copying them and we'll soon see that it's actually copying them so for this second row let me just go back down here okay so additionally we perform an interventional experiment to investigate the extent to which amputees have actually learned the causal mechanism underlying the lookup task and if you see these two keywords and if you're familiar with Judea Pearl's work you'll notice that this is basically rank two experiments so we have this thing called letter of causation the first letter is just observing data the second letter would be having acting in the environment or doing these these interventional kinds of experiments where you're setting something to certain value and then observing and finally we have a counterfactual level where we are imagining like different worlds and that sounds super abstract for this context but yeah just just I wanted to just associate this sentence here with Judea Pearl's work so if you want to know more about it you can check some of my previous LinkedIn posts or you can read the book of why from from Judea Pearl as a nice introduction to his work so the model is not confronted with target values associated with features that are highly unlikely under the training data and let's see why that is so basically what I do as you can see so it's the same experiment as the above but this time they take this target value and instead of basically keeping it the way it was in the original data set they tweak it they take that maybe there was 1.3 and they're going to kind of mess it up and put 7.5 here and that combination 7.5 with similar attributes have never been seen in the training data set so now if the model learned how to just kind of use the attributes and figure out the target value is gonna fail here and on the other hand if you'd learn how to do what I previously mentioned and that's just to attend to the vector and copy that value it's gonna have nice performance and as we can see here the Pearson correlation is again great that means it learned how to no matter these interventions it learned how to predict the exact value from from these slots and that means it learned the correct relational mechanism that's the point of these experiments and so we now confidently conclude that the MPTs robustly learn the causal data generating mechanism underlying the semi-synthetic data set this requires MPTs to learn a non-trivial sequence of computational steps they must learn to match rows based on similarity of relevant features to look up the target value of the duplicated data point and to copy that value into the target of the mask data point so that's what I already explained and yeah hopefully now it makes more sense okay next experiment they did was to corrupt the data points and see whether it's failing because it learned how to actually rely on those other data points so what I do they said here so this completely corrupts information from all data points except except the one for which we evaluate hence a model that relies meaningfully on attention between data points will show deteriorating performance so let me just explain you what they actually did and they did the following thing so we have a data set here and let's say we have a bunch of data points and let's say we want to predict this value here okay so this is the data point for which we want to figure out predict the class what I do is they just shuffle all of these attributes of all of the others so all of these other data points are going to get shuffled and that means this feature vector can't be matched now with other feature vectors in them like using the we can't use the mechanism we just described it's using and that's just figuring out matching the vector finding maybe I don't know maybe it learned to do cosine similarity now I can't do that because we shuffle the attributes and what I show is that performance deteriorates which is a further proof that it learned the correct relational mechanism it appears that when MPTs do not find it advantageous to rely on attention between data points during training they can learn to completely ignore the other inputs essentially collapsing into a standard parametric model this supports our earlier claims that amputees can learn end-to-end from data the extent to which they rely on other data points for prediction so that's a fancy way of saying that basically sometimes if we have a data set again so sometimes it's actually just relying on this particular row on a single data point to predict this value here because for that task you figure out it doesn't actually need to use other data points and for those data sets the the performance they show does not deteriorate which is again a further proof about proof informal proof that basically it learned the correct mechanism here they just have a table a bunch of different data sets you can see regressions here so yeah on certain data sets as I just mentioned it's not deteriorating that much because it doesn't rely on other data points that much okay having said that let me see if we have something else interesting and yeah this is the last part that's super interesting and they now try to figure out how the model finds those similar vectors so and what I do is we sort the input data with respect to their feature space it's actually input space as I saw in the app appendix but maybe the authors can correct me here distance such that the similar data points are now close to each other and so let me explain what that means basically that means you have the original tabular data set you have your data points what I do is they find maybe using L2 metric define the closest vectors here and they're just going to sort them so now we have a vector here and a vector here and these two have really small L2 distance and now we have you know vector here and these two have really small L2 distance which that means that basically these two also have maybe a bit bigger L2 distance okay so you get the point basically we're slowly adding stacking vectors which are further away looking at the L2 distance and when we do that and we apply NPT and when we look at the certain like attention patterns from a certain like head we get this pattern and this means that is basically attending to the vectors which are basically close in the input space which means it's attending to similar vectors according to this L2 distance and if you don't know how to parse this diagram let me help you basically if we have if this is row I this row is going to attend and assume we have a sort of table so this table is sorted according to L2 so let's say this is I this is vector I and it's somewhere in the middle of the table and that vector is going to attend to its proximity so it's going to attend to these vectors really strongly and then it's going to drop off and then it's going to have really weak attendance to maybe this vector here so it's going to tend to these vectors really strongly and the strength will kind of fade away going downwards or upwards here and that's exactly what you see in this attention map basically if this is vector I and this is just n number of data points this is just n number of data points this means that this vector as you can see attends to itself really strongly and to its neighborhood but then it deteriorates because you can see that the dark blue means the attention coefficients are really really really weak there okay that's the whole point and yeah further they had some additional experiment where they're doing similar thing but the conclusion was the same basically MPT learns how to attend to those vectors which have which are really close in the input space looking at the L2 distance okay that's pretty much it they dimension some problems like these scaling limitations which I have already flagged and that's pretty much it so we saw that MPT's learn how to do relational reasoning and because they learn the correct mechanism on that simple semi-synthetic task it can generalize much better than if they actually learn how to deduce the value just looking at the attributes and instead they learn how to just copy paste the values from the corresponding call of Rose and yeah hopefully this was useful if you found this video useful share it out and subscribe and until next time bye bye
[{"start": 0.0, "end": 3.7600000000000002, "text": " What's up? In this video I'm covering this new exciting paper called"}, {"start": 3.7600000000000002, "end": 8.72, "text": " Self-Attention Between Data Points Going Beyond Individual Input-Output Pairs in Deep Learning,"}, {"start": 9.44, "end": 14.96, "text": " which introduces this model called non-parametric transformers. And now, non-parametric doesn't"}, {"start": 14.96, "end": 19.92, "text": " mean it doesn't have parameters, it's quite the opposite. So it just means that it doesn't scale"}, {"start": 19.92, "end": 25.28, "text": " well with data. Basically, the more data you give it, the more parameters it's going to need."}, {"start": 25.28, "end": 31.12, "text": " And in general, these non-parametric models either have this time-wise or space-wise slash"}, {"start": 31.12, "end": 38.400000000000006, "text": " memory-wise dependency on the number of data points. So having said that, the main idea"}, {"start": 38.400000000000006, "end": 44.08, "text": " this paper introduces is that it's a kind of attempt to generalize even transformers."}, {"start": 44.08, "end": 51.6, "text": " So whereas transformers basically assume there is a fully connected graph between the input tokens,"}, {"start": 51.6, "end": 57.04, "text": " and so they kind of tend to everything, here they additionally actually attend over different data"}, {"start": 57.04, "end": 63.28, "text": " points. So that means it's assuming an implicit fully connected graph over the data points."}, {"start": 63.28, "end": 70.48, "text": " And I'll get into those details in a moment. And basically, even during the inference,"}, {"start": 70.48, "end": 77.52000000000001, "text": " they need the training data points, and they'll use those to better infer the class or the"}, {"start": 77.52, "end": 82.47999999999999, "text": " value if you're doing regression of the test data point. And we'll see all of those details."}, {"start": 82.47999999999999, "end": 88.39999999999999, "text": " So that's like the main innovation that this paper brings. They get a really nice results"}, {"start": 88.39999999999999, "end": 94.24, "text": " on some tabular data sets, which is something that previous methods like XGBoost were dominating."}, {"start": 94.24, "end": 99.28, "text": " And yeah, that's pretty exciting. So having said all that, let me jump into the details of the"}, {"start": 99.28, "end": 105.84, "text": " paper. They say here, we challenge a common assumption, underlying most supervised deep"}, {"start": 105.84, "end": 110.0, "text": " learning, that our model makes a prediction depending only on its parameters and the features"}, {"start": 110.0, "end": 114.72, "text": " of a single input. To this end, we introduce a general purpose deep learning architecture"}, {"start": 114.72, "end": 119.84, "text": " that takes as input the entire data set, instead of processing one data point at a time. Our"}, {"start": 119.84, "end": 126.32000000000001, "text": " approach uses self attention to reason about relationships between data points explicitly,"}, {"start": 126.32000000000001, "end": 131.12, "text": " which can be seen as realizing non-parametric models using parametric attention mechanism."}, {"start": 131.12, "end": 136.96, "text": " And again, you can see here, so entire data set, they are learning from the entire data set,"}, {"start": 136.96, "end": 144.56, "text": " and they're doing inference, as we'll see soon see on the pretty much, because it doesn't scale well"}, {"start": 144.56, "end": 149.28, "text": " with data, they'll have some simplifying assumptions and approximations. So they'll be using some chunk"}, {"start": 149.28, "end": 155.76, "text": " of the training data to do inference on. As I said, the model is called non-parametric transformers."}, {"start": 155.76, "end": 165.12, "text": " And so a quick note here, like just on like conventional non-parametric models. So they say"}, {"start": 165.12, "end": 169.28, "text": " here, conventional non-parametric models cannot learn in the sense familiar to deep learning"}, {"start": 169.28, "end": 174.64, "text": " practitioners. Interactions from the data limiting the flexibility these models have in adapting to"}, {"start": 174.64, "end": 181.28, "text": " the data at hand. And the good example would be a KNN, K nearest neighbors algorithm. So just a quick"}, {"start": 181.28, "end": 186.8, "text": " note how it works, and you'll then understand what non-parametric models mean. So let's say we have"}, {"start": 186.8, "end": 192.16, "text": " some classes here, like maybe we have two classes, one is like triangles, and then maybe we have"}, {"start": 192.16, "end": 198.72, "text": " something like circles, something like this. So and assume, so this is your data set, and now assume"}, {"start": 198.72, "end": 203.36, "text": " I give you a query point and ask you what's the class of this data point. And now if we assume"}, {"start": 203.36, "end": 210.96, "text": " that K equals three, basically that means we'll find the three nearest neighbors, hence the K"}, {"start": 210.96, "end": 216.48000000000002, "text": " nearest neighbors algorithm. So that's going to be this, and then we'll output by majority voting,"}, {"start": 216.48000000000002, "end": 222.72000000000003, "text": " let's take that simple variant of the algorithm without including the actual distances, and"}, {"start": 223.28000000000003, "end": 228.48000000000002, "text": " either way we're going to have triangle here. So as you can see, the more data you have here, so if"}, {"start": 228.48, "end": 236.72, "text": " we had a bunch of different circles and triangles, and if K goes to infinity, and also assuming data"}, {"start": 236.72, "end": 241.28, "text": " goes to infinity, that means that this algorithm doesn't scale well, because you'll have to search"}, {"start": 241.28, "end": 248.16, "text": " for like infinite number of nearest neighbors. And so yeah, contrast that to maybe a neural network"}, {"start": 248.16, "end": 253.92, "text": " where no matter the like the number of data points, the size of your data set, you'll have"}, {"start": 253.92, "end": 259.36, "text": " a specified pre-specified architecture whose parameters won't change, and the inference time"}, {"start": 259.36, "end": 264.24, "text": " won't change, because you always have to pass a single data point or a batch whatever through the"}, {"start": 264.24, "end": 269.68, "text": " neural network. So that's the difference between parametric and non-parametric, and as we'll soon"}, {"start": 269.68, "end": 274.64, "text": " see, now we'll see how the model actually works. So I first want to explain you the architecture"}, {"start": 274.64, "end": 282.64, "text": " itself and the data layout. So you can see here, their data set is tabular, so they basically assume"}, {"start": 282.64, "end": 288.56, "text": " they concentrate on like on this single target regression and classification. That means the"}, {"start": 288.56, "end": 292.24, "text": " following. That means they are placing the attributes along, so these are data points,"}, {"start": 292.24, "end": 297.52, "text": " as you can see, along this axis. We have here attributes, but in this last column, they're"}, {"start": 297.52, "end": 301.44, "text": " going to assume they're having like targets, either classification targets, like what's the"}, {"start": 302.08, "end": 308.24, "text": " the class for these. So for this data point here, we have attributes, and this is going to be some"}, {"start": 308.24, "end": 313.04, "text": " some like class, okay? Or if we're doing regression, there's going to be some continuous value."}, {"start": 313.04, "end": 320.96000000000004, "text": " So that's just how they structure it, and as you can see here, if we had a parametric model,"}, {"start": 320.96000000000004, "end": 325.6, "text": " maybe a transformer, this is how the inference would look like, except for the transformer,"}, {"start": 325.6, "end": 330.64, "text": " you'd have edges going here as well, and here as well, because we are attending, every token is"}, {"start": 330.64, "end": 336.48, "text": " attending to every token, i.e. every attribute is attending to every attribute. So finally, we output"}, {"start": 336.48, "end": 342.64000000000004, "text": " the the target value, whatever that is, either class or a continuous value. And here in MPT,"}, {"start": 342.64000000000004, "end": 348.40000000000003, "text": " as you can see, they're also leveraging other data points, so maybe some training data points,"}, {"start": 349.28000000000003, "end": 355.44, "text": " and basically using all that information, they can reason and figure out this value, and they show"}, {"start": 355.44, "end": 362.0, "text": " that this inductive bias is really useful, so having the ability to attend to other data points"}, {"start": 362.0, "end": 366.72, "text": " as well. So that's an interesting shifting perspective, because usually we're used to"}, {"start": 367.44, "end": 374.64, "text": " this setup, but yeah, that's cool. Okay, so let me start explaining how the thing works, and"}, {"start": 375.28, "end": 380.96, "text": " first things first, like the objective is the following. So MPTs, so non-parametric transformers,"}, {"start": 380.96, "end": 384.8, "text": " training objective is to reconstruct a corruptive version of the input data set."}, {"start": 384.8, "end": 390.48, "text": " Similar to BERT, we apply stochastic masking to both features and targets, and minimize a loss"}, {"start": 390.48, "end": 397.68, "text": " on MPTs prediction at entries masked out in the input. So basically, they'll have a binary mask"}, {"start": 398.64000000000004, "end": 404.96000000000004, "text": " telling them which parts are masked here, and those will be predicted. And a quick side note,"}, {"start": 405.6, "end": 412.16, "text": " if you're not familiar with BERT, it's very simple, it's a famous transformer, and the way it works,"}, {"start": 412.16, "end": 416.72, "text": " the way they train BERT is the following. So you have input tokens, so this is some sentence,"}, {"start": 416.72, "end": 422.64000000000004, "text": " like help print Mayuko, and by the way, this is from JLMR's blog, I'll link it down in the description,"}, {"start": 422.64000000000004, "end": 427.84000000000003, "text": " and what they do when they train BERT is they mask out certain tokens. So this Mayuko"}, {"start": 427.84000000000003, "end": 434.72, "text": " will be masked by like a special symbol called mask, whatever, with some probability, or they'll"}, {"start": 434.72, "end": 440.96000000000004, "text": " just place some random value there, okay? And they do that for a number of tokens, and now the goal is"}, {"start": 440.96, "end": 448.4, "text": " to output Mayuko here, so output the correct class up here, so if they masked some token here, they'll"}, {"start": 448.4, "end": 453.52, "text": " have to output it here. So the final loss will just be, if you're doing classification,"}, {"start": 455.84, "end": 461.84, "text": " like they'll just do like basically cross entropy loss here, and they're gonna back prop that and"}, {"start": 461.84, "end": 468.08, "text": " train the model. So it's a self-supervised way of training, and I'm just ignoring, for the time being,"}, {"start": 468.08, "end": 472.24, "text": " for those of you who know how BERT works, I'm just ignoring this part of next sentence prediction,"}, {"start": 473.03999999999996, "end": 478.47999999999996, "text": " and focusing on these, because that's the part that's really similar in MPTs, they're doing the"}, {"start": 478.47999999999996, "end": 486.96, "text": " same thing as BERT, pretty much. Okay, so they also assume, as I said, that the targets are the final"}, {"start": 486.96, "end": 492.96, "text": " attribute of this table. So as I said, they're gonna mask out these values, and they're gonna try"}, {"start": 492.96, "end": 498.0, "text": " predict those values, but they're also going to mask out some of the attributes as well, so you'll"}, {"start": 498.0, "end": 505.6, "text": " see, that's a combination. Let's first focus on the high-level picture. So this is, as I said,"}, {"start": 505.6, "end": 511.2, "text": " this is the input, so this is your tabular data set, you have n data points, you have d attributes,"}, {"start": 511.2, "end": 514.88, "text": " and some of the values are masked, so either attributes or the target values."}, {"start": 515.76, "end": 523.6, "text": " So what I do is, they first embed this data into this n, d, and now we have dimension e here,"}, {"start": 523.6, "end": 527.9200000000001, "text": " and I'll get into a bit more details of how exactly the input and output embeddings work,"}, {"start": 527.9200000000001, "end": 533.2, "text": " but having done that, somehow, consider it a black box for now, what I do is the following."}, {"start": 533.2, "end": 539.0400000000001, "text": " So they apply, so they first flatten out this thing, and they apply a data point, so basically"}, {"start": 539.36, "end": 543.76, "text": " the same thing as what was finally did in the original transformer, so we have self-attention"}, {"start": 543.76, "end": 549.28, "text": " from the original transformer applied here, across data points first. So they basically attend,"}, {"start": 549.28, "end": 554.64, "text": " as you can see these lines here, every data point attends to every other data point, so we do"}, {"start": 554.64, "end": 560.3199999999999, "text": " a VASVANI type self-attention, then they reshape it, and then they do the same thing just across"}, {"start": 560.3199999999999, "end": 568.64, "text": " the, this time across attributes, and they repeat this n times, basically as a for loop, and then"}, {"start": 568.64, "end": 575.68, "text": " they didn't draw it here, but basically they have, again, they have output linear projection layer,"}, {"start": 575.68, "end": 580.4799999999999, "text": " and that's the whole thing, the architecture is pretty simple. Now if you're familiar with"}, {"start": 580.4799999999999, "end": 585.52, "text": " recently published MLP mixer, and I'll link it somewhere here if you haven't watched my video,"}, {"start": 585.52, "end": 593.5999999999999, "text": " I've explained it there, but like, you can see that if we treat patch like a data point, so MLP"}, {"start": 593.5999999999999, "end": 598.8, "text": " mixer like inputs an image, you split it into patches, and if we treat those patches as data"}, {"start": 598.8, "end": 604.3199999999999, "text": " points, there is a similarity in the mixing part at least, so they also have, so basically"}, {"start": 604.32, "end": 609.6, "text": " they also have the first part where they are attending over different channels, over different"}, {"start": 609.6, "end": 614.32, "text": " patches, so different data points, but the same channel, and here they have the part"}, {"start": 614.32, "end": 618.1600000000001, "text": " where they are attending basically the samehow where they are attending basically the same"}, {"start": 618.1600000000001, "end": 621.9200000000001, "text": " patch, but different channels, so that will be, that will correspond to different attributes"}, {"start": 621.9200000000001, "end": 626.32, "text": " here. So this corresponds to this part, and this part where they are attending over different"}, {"start": 626.32, "end": 631.12, "text": " data points corresponds to this part, where they are attending over different patches."}, {"start": 631.12, "end": 635.14, "text": " they're attending over different patches. So I just thought that's a neat"}, {"start": 635.14, "end": 639.46, "text": " analogy and other than that they are not using attendance. So the whole point of"}, {"start": 639.46, "end": 644.0600000000001, "text": " MLP mixture paper was to ditch the attention and just focus on the MLPs so"}, {"start": 644.0600000000001, "end": 647.92, "text": " I have just have less even less inductive biases in this architecture."}, {"start": 647.92, "end": 651.22, "text": " Okay there was a small digression hopefully you found that connection"}, {"start": 651.22, "end": 657.66, "text": " useful. Now let me go and try and focus on explaining the the actual architecture."}, {"start": 657.66, "end": 662.48, "text": " So quick mention again we focus on single target classification and"}, {"start": 662.48, "end": 669.48, "text": " regression corresponding to a masking matrix M with ones at all entries of the"}, {"start": 669.48, "end": 676.04, "text": " table column X that this is just a means that the last column is masked but"}, {"start": 676.04, "end": 680.48, "text": " online multi-target settings imputation self-supervision using input feature is"}, {"start": 680.48, "end": 684.56, "text": " also possible. So that's what I told you when I said like basically they can also"}, {"start": 684.56, "end": 689.7199999999999, "text": " mask some other parts of the of the input data and that's really cool. So the cool"}, {"start": 689.7199999999999, "end": 695.16, "text": " part here is that and I noted it here basically it joins the learns imputation"}, {"start": 695.16, "end": 699.3199999999999, "text": " plus prediction. So in conventional approaches what you usually have to do"}, {"start": 699.3199999999999, "end": 703.16, "text": " when you use those boosting methods you have to have a two-stage approach. The"}, {"start": 703.16, "end": 707.8, "text": " first stage just imputes the values and now again imputation just means that"}, {"start": 707.8, "end": 712.04, "text": " you're missing some values. If this is your tabular data and let's assume"}, {"start": 712.04, "end": 716.3199999999999, "text": " you're missing some value and I'll just denote that as a question mark these"}, {"start": 716.3199999999999, "end": 721.4599999999999, "text": " algorithms usually have to maybe apply some heuristic like maybe they'll have to"}, {"start": 721.4599999999999, "end": 725.8399999999999, "text": " attend over the same row and maybe find I don't know like mean or median and"}, {"start": 725.8399999999999, "end": 730.68, "text": " impute it right here or they can even train a dedicated machine learning"}, {"start": 730.68, "end": 735.3199999999999, "text": " model which will learn how to impute those values. Once the imputation is done"}, {"start": 735.3199999999999, "end": 740.4399999999999, "text": " that stage is done now they only can do prediction. Here this model MPT does this"}, {"start": 740.44, "end": 744.5200000000001, "text": " jointly and that's really cool because it can learn much better how to do"}, {"start": 744.5200000000001, "end": 751.0, "text": " imputation in this end-to-end deep learning scenario because basically it"}, {"start": 751.0, "end": 755.2800000000001, "text": " can also attend other data points and make much better much more educated"}, {"start": 755.2800000000001, "end": 761.0400000000001, "text": " decisions here. So that's something I thought mentioning and yeah I'll get"}, {"start": 761.0400000000001, "end": 764.8800000000001, "text": " back to this equivariance thing and I'll make some nice connections to graph"}, {"start": 764.8800000000001, "end": 769.4000000000001, "text": " neural networks but like for the time being let me just again I showed you the"}, {"start": 769.4, "end": 772.24, "text": " the high-level picture of how the architecture works now let me get into a"}, {"start": 772.24, "end": 777.36, "text": " bit more details. So again I won't be explaining the actual self-attention"}, {"start": 777.36, "end": 780.92, "text": " mechanism I have a video I made a video on that you can check it out it's a"}, {"start": 780.92, "end": 784.52, "text": " basically Vasvani self-attention so you've been seeing if you're following"}, {"start": 784.52, "end": 789.1999999999999, "text": " your like recent research this is being used everywhere so I'll skip that part"}, {"start": 789.1999999999999, "end": 794.56, "text": " and I'll just focus on explaining the input output embeddings. So let me just"}, {"start": 794.56, "end": 800.7199999999999, "text": " scroll down in the appendix okay it's gonna be so okay by the way it's the"}, {"start": 800.7199999999999, "end": 805.68, "text": " paper is really neat they have they added really cool things like checklists"}, {"start": 805.68, "end": 811.56, "text": " and computational requirements and also some implicated like societal"}, {"start": 811.56, "end": 815.5999999999999, "text": " implications so I really love how the paper is written and yeah kudos to the"}, {"start": 815.5999999999999, "end": 821.4, "text": " authors. Okay let's see how the how the thing works in the initial stage of the"}, {"start": 821.4, "end": 826.24, "text": " pipeline of the architecture so basically we have we have a data set so"}, {"start": 826.24, "end": 832.12, "text": " this is the data set tabular so we have and data points we have the attributes"}, {"start": 832.12, "end": 837.92, "text": " one of them is also the the target column so implicitly that's the last"}, {"start": 837.92, "end": 842.8, "text": " column and how it works is the following so they say it here we'll learn separate"}, {"start": 842.8, "end": 847.24, "text": " embedding weights for each attribute and that makes sense because each attribute"}, {"start": 847.24, "end": 853.88, "text": " contains its own semantics and we'll now see like even like in more details why"}, {"start": 853.88, "end": 859.04, "text": " that is so let's say we have this column so let's assume this this column"}, {"start": 859.04, "end": 863.4, "text": " contains categorical variables so we'll have a certain number of classes there"}, {"start": 863.4, "end": 867.6, "text": " and the first step they do for a categorical variable is they just"}, {"start": 867.6, "end": 874.04, "text": " basically encoded into a one-hot encoding so this is going to be size of C"}, {"start": 874.04, "end": 880.0799999999999, "text": " where C is the number of classes and basically so so let's say if we if you"}, {"start": 880.0799999999999, "end": 884.24, "text": " have a number I don't know two and assume we have I don't know maybe four"}, {"start": 884.24, "end": 889.7199999999999, "text": " classes this is going to be embedded into a one-hot vector like this so"}, {"start": 889.7199999999999, "end": 896.68, "text": " basically we have four slots here and one goes here zeros elsewhere okay so"}, {"start": 896.68, "end": 902.5999999999999, "text": " they're gonna embed every single attribute in that column across"}, {"start": 902.6, "end": 907.5600000000001, "text": " different data points into that one-hot embedding okay so the next step they do"}, {"start": 907.5600000000001, "end": 914.64, "text": " is they append the the mask here so they'll append the mask down all the way"}, {"start": 914.64, "end": 919.72, "text": " down right and they do this for every single column so now let's assume this"}, {"start": 919.72, "end": 925.0400000000001, "text": " one is maybe continuous variable the first thing they do here is they"}, {"start": 925.0400000000001, "end": 929.8000000000001, "text": " standardize the column so they just make it have mean of zero and standard"}, {"start": 929.8, "end": 936.0799999999999, "text": " standard deviation of one or variance equal to one and basically now after"}, {"start": 936.0799999999999, "end": 940.64, "text": " they do that they'll just have a single dimension here so that's one not C and"}, {"start": 940.64, "end": 946.0799999999999, "text": " they'll append the mask again okay so that's gonna be two and as you can see"}, {"start": 946.0799999999999, "end": 950.3599999999999, "text": " and also some different column may have different number of classes here so it"}, {"start": 950.3599999999999, "end": 957.1999999999999, "text": " may be C C prime whatever and that means for that reason we'll basically have to"}, {"start": 957.2, "end": 961.96, "text": " have different independent linear projection layers not just because of"}, {"start": 961.96, "end": 967.24, "text": " the semantics but also because of the pure shape mismatch so you can think of"}, {"start": 967.24, "end": 972.08, "text": " it like both ways so once we have that we just have like linear projection"}, {"start": 972.08, "end": 975.9200000000001, "text": " layer so which will just project all of these so whatever the number of"}, {"start": 975.9200000000001, "end": 980.8000000000001, "text": " dimensions here is into e dimensions so this is going to be kind of projected"}, {"start": 980.8000000000001, "end": 986.2, "text": " into like you can imagine having a fully connected layer here blah blah I don't"}, {"start": 986.2, "end": 990.2, "text": " know how to draw today but basically you can imagine it like this is you'll have"}, {"start": 990.2, "end": 999.0, "text": " e dimensions and we'll end up with a n times d times e volume the same thing"}, {"start": 999.0, "end": 1004.6800000000001, "text": " you saw on the high level diagram so that's the embedding part almost let me"}, {"start": 1004.6800000000001, "end": 1009.6, "text": " just show you the formula so this in code this is the part that embeds the"}, {"start": 1009.6, "end": 1014.08, "text": " categorical variables into one hot embeddings then they append the mask so"}, {"start": 1014.08, "end": 1021.4000000000001, "text": " again the mask will just have dimensions n times d and if you have one somewhere"}, {"start": 1021.4000000000001, "end": 1025.48, "text": " and I forgot to mention that part if you have one somewhere that means that part"}, {"start": 1025.48, "end": 1031.72, "text": " is going to be masked out so let's assume this part here has one so that"}, {"start": 1031.72, "end": 1039.44, "text": " means they do the following aside from appending one here at the last like"}, {"start": 1039.44, "end": 1044.8400000000001, "text": " slot here they're going to mask out all of this with high probability so all of"}, {"start": 1044.8400000000001, "end": 1051.52, "text": " these will get set to zero and so that means that that one hot vector will just"}, {"start": 1051.52, "end": 1057.04, "text": " be zeroed out so this hot this one hot vector will just be zeroed out so we'll"}, {"start": 1057.04, "end": 1061.4, "text": " have just all zeros all the way here so as in birth this is either going to be"}, {"start": 1061.4, "end": 1066.56, "text": " zeroed out or they're gonna just set our random value in there and that's how the"}, {"start": 1066.56, "end": 1070.8799999999999, "text": " thing works so again getting back to to the formula you can see after we do the"}, {"start": 1070.8799999999999, "end": 1074.36, "text": " one hot encoding for categorical variables or just normalization for"}, {"start": 1074.36, "end": 1079.6799999999998, "text": " continuous variables we append the mask we project this into the into this e"}, {"start": 1079.6799999999998, "end": 1085.6799999999998, "text": " dimensions and then we just concatenate across all the different columns IE"}, {"start": 1085.6799999999998, "end": 1090.32, "text": " attributes and then they additionally have these learned embeddings for"}, {"start": 1090.32, "end": 1094.3999999999999, "text": " index and type I'm not sure whether they did ablation on whether they actually"}, {"start": 1094.4, "end": 1098.5600000000002, "text": " need the both of these whether it can be jointly learned like the type and index"}, {"start": 1098.5600000000002, "end": 1102.72, "text": " but yeah if anybody knows or the authors are watching the video they can maybe"}, {"start": 1102.72, "end": 1107.66, "text": " explain it all in the comments having done that for a single attribute column"}, {"start": 1107.66, "end": 1110.8000000000002, "text": " so this is for a single attribute column so this was this part maybe we're"}, {"start": 1110.8000000000002, "end": 1115.3200000000002, "text": " focusing only on this specific column on this volume now they just concatenate"}, {"start": 1115.3200000000002, "end": 1121.4, "text": " all of those as you can see across the axis D so this is the D axis and that's"}, {"start": 1121.4, "end": 1131.0800000000002, "text": " it that's it you get the NDE volume and similarly for the output embedding I'll"}, {"start": 1131.0800000000002, "end": 1135.52, "text": " just briefly explain it so here they're not predicting the mass variable so"}, {"start": 1135.52, "end": 1142.4, "text": " they'll just end up with so EJ was the notation they used to denote this part"}, {"start": 1142.4, "end": 1150.1200000000001, "text": " like without the the masking dimension that's added so they're just going to"}, {"start": 1150.12, "end": 1159.3999999999999, "text": " again linearly project all of these this E into into E let's call it subscript"}, {"start": 1159.3999999999999, "end": 1164.56, "text": " subscript D where this is basically depends on the on the column so for this"}, {"start": 1164.56, "end": 1170.52, "text": " column this ED is going to be equal to C for this continuous variable column is"}, {"start": 1170.52, "end": 1177.1999999999998, "text": " going to be equal to 1 etc so yeah that's basically needed so that we can"}, {"start": 1177.2, "end": 1183.24, "text": " then do the like appropriate apply the appropriate loss across that column so"}, {"start": 1183.24, "end": 1189.28, "text": " it's very easy they say here so it's one for continuous attributes it's the"}, {"start": 1189.28, "end": 1193.04, "text": " number of categories for a categorical attribute and to obtain the final"}, {"start": 1193.04, "end": 1198.2, "text": " prediction matrix we take the argmax over the categorical predictions so"}, {"start": 1198.2, "end": 1202.24, "text": " that's how they do the prediction so basically whatever they spit out in the"}, {"start": 1202.24, "end": 1207.16, "text": " last layer maybe like they'll have some probabilities obviously if we extract"}, {"start": 1207.16, "end": 1212.1200000000001, "text": " this part here and then just make it a bit nicer to see so if I extract it here"}, {"start": 1212.1200000000001, "end": 1218.76, "text": " it's going to be some distribution and like dimensions here are going to be C"}, {"start": 1218.76, "end": 1225.08, "text": " as I said so it's gonna be just some distribution of across the the classes"}, {"start": 1225.08, "end": 1230.68, "text": " and they're just gonna do like like normal like argmax and find the the the"}, {"start": 1230.68, "end": 1234.44, "text": " highest probability class and that's their prediction okay that was lots of"}, {"start": 1234.44, "end": 1239.4, "text": " details on the on the architecture let me let me get back to the paper and tell"}, {"start": 1239.4, "end": 1245.64, "text": " you some interesting stuff so first things first I don't want to finish with"}, {"start": 1245.64, "end": 1250.6000000000001, "text": " that with the loss so as I said before so they're applying stochastic feature"}, {"start": 1250.6000000000001, "end": 1254.2, "text": " masking they're applying stochastic target masking similarly to BERT and"}, {"start": 1254.2, "end": 1258.8, "text": " during the training we compute the negative log likelihood loss at training"}, {"start": 1258.8, "end": 1263.8799999999999, "text": " targets as well as the auxiliary loss from masked out features so that's the"}, {"start": 1263.8799999999999, "end": 1268.6, "text": " part I mentioned so assuming we have a classification problem what I do is the"}, {"start": 1268.6, "end": 1274.68, "text": " following so they'll output again this would be the output of their model of"}, {"start": 1274.68, "end": 1280.96, "text": " the NPT and let's focus on a single attribute on a single slot here"}, {"start": 1280.96, "end": 1285.56, "text": " basically they said this is going to be C and they're simply going to take the"}, {"start": 1285.56, "end": 1290.0, "text": " the ground truth position so maybe this is the ground truth class they're gonna"}, {"start": 1290.0, "end": 1294.04, "text": " take the probability that they had there so let me just kind of draw it"}, {"start": 1294.04, "end": 1298.12, "text": " three-dimensional here so they're just going to extract the probability there"}, {"start": 1298.12, "end": 1306.9199999999998, "text": " and they're just gonna do minus log p of I so in order to minimize the loss you"}, {"start": 1306.9199999999998, "end": 1311.44, "text": " want to have your obviously so that's like cross entropy simple stuff you"}, {"start": 1311.44, "end": 1317.3400000000001, "text": " want to have your probability going to one this is the output PI and this is"}, {"start": 1317.3400000000001, "end": 1322.76, "text": " the loss okay the y-axis so they just summed that out across all of the"}, {"start": 1322.76, "end": 1328.3200000000002, "text": " masked out features so assume we had one here for for mask assume we had maybe if"}, {"start": 1328.3200000000002, "end": 1334.18, "text": " we had one here we're gonna additionally extract the class for that one and do"}, {"start": 1334.18, "end": 1339.52, "text": " minus log PI and we're gonna just do a sum over those and just do a back prop"}, {"start": 1339.52, "end": 1343.44, "text": " through the model hopefully that was clear so long sure store they're just"}, {"start": 1343.44, "end": 1349.68, "text": " going to sum up across all of the masked by their features or targets here and"}, {"start": 1349.68, "end": 1354.6399999999999, "text": " they're going to do a sum and just do a big prop on their loss okay so"}, {"start": 1354.6399999999999, "end": 1358.68, "text": " interestingly stochastic target masking means that many training targets are"}, {"start": 1358.68, "end": 1362.56, "text": " unmasked to the model at training time so this is interesting that means that"}, {"start": 1362.56, "end": 1367.36, "text": " this column which contains the targets will not be like completely massive some"}, {"start": 1367.36, "end": 1371.1999999999998, "text": " parts will have zeros here and that means that the model can actually see"}, {"start": 1371.1999999999998, "end": 1376.36, "text": " that the correct like target value for that for that data point this allows"}, {"start": 1376.36, "end": 1381.3999999999999, "text": " MPTs to learn to predict at each epoch the masked targets of certain training"}, {"start": 1381.3999999999999, "end": 1386.52, "text": " data points using the targets of other training data points in addition to all"}, {"start": 1386.52, "end": 1390.76, "text": " training data features okay so that's an interesting part and we'll see how that"}, {"start": 1390.76, "end": 1397.24, "text": " helps them do relational reasoning later on here's just a short note I'll"}, {"start": 1397.24, "end": 1402.1200000000001, "text": " obviously this approach will have some problems with scaling because they are"}, {"start": 1402.1200000000001, "end": 1406.4, "text": " attending like bunch of training data points during inference as well and yeah"}, {"start": 1406.4, "end": 1410.1200000000001, "text": " that's something that future research will need to tackle with okay"}, {"start": 1410.1200000000001, "end": 1415.56, "text": " architecture explained now let me actually show you some exciting results"}, {"start": 1415.56, "end": 1420.24, "text": " and first give you some connections to graph neural networks that I promised"}, {"start": 1420.24, "end": 1423.48, "text": " here so first things first they sit here"}, {"start": 1423.48, "end": 1428.52, "text": " MPTs are equivalent to a permutation of the data points in other words if the"}, {"start": 1428.52, "end": 1433.08, "text": " set of input data points are shuffled so if we have the tabular data you have"}, {"start": 1433.08, "end": 1440.92, "text": " and and and rows d columns so we kind of shuffle these rows MPTs produce the same"}, {"start": 1440.92, "end": 1446.04, "text": " predictions but shuffled in an analogous manner so that's the equivalent part so"}, {"start": 1446.04, "end": 1450.56, "text": " it should not depend on their ordering that's the important part so now let me"}, {"start": 1450.56, "end": 1454.96, "text": " just kind of give you some connections with CNN's and graph neural networks and"}, {"start": 1454.96, "end": 1460.04, "text": " hopefully you'll find that useful okay where are we let me just find the yep"}, {"start": 1460.04, "end": 1466.1599999999999, "text": " here it is so they mentioned here that MPTs can be seen as a conceptual"}, {"start": 1466.1599999999999, "end": 1469.96, "text": " generalization of graph convolutional networks and again I've covered that in"}, {"start": 1469.96, "end": 1472.96, "text": " my previous video you can check it out I'll link it somewhere here and the"}, {"start": 1472.96, "end": 1476.56, "text": " other graph neural networks like get again I've checked I've covered that one"}, {"start": 1476.56, "end": 1481.0, "text": " as well so that's graph attention network engine in which a set of"}, {"start": 1481.0, "end": 1486.1599999999999, "text": " dependencies edges between data points is not known a priori and instead is"}, {"start": 1486.1599999999999, "end": 1491.44, "text": " learned from data using self-attention so this is the interesting part and now"}, {"start": 1491.44, "end": 1495.56, "text": " let me let me let me show you what they mean by that so amputees learn an"}, {"start": 1495.56, "end": 1499.6, "text": " implicit graph structure so whereas if you have if you're familiar with graph"}, {"start": 1499.6, "end": 1503.52, "text": " neural networks let me let me draw a graph here basically you have some notes"}, {"start": 1503.52, "end": 1510.8, "text": " in your graph and for example for forget you'll have explicit connections here"}, {"start": 1510.8, "end": 1515.6399999999999, "text": " okay you'll have something like this and then what graph attention network will"}, {"start": 1515.6399999999999, "end": 1520.6399999999999, "text": " do it will just kind of assign certain attention coefficients to each of these"}, {"start": 1520.6399999999999, "end": 1526.48, "text": " edges and it's going to aggregate all of these node features so every node has a"}, {"start": 1526.48, "end": 1532.16, "text": " feature vector associated with it it's going to kind of aggregate all of these"}, {"start": 1532.16, "end": 1540.0, "text": " by some like invariant like mapping and they're going to get the final embedding"}, {"start": 1540.0, "end": 1544.24, "text": " for this vector so this was maybe so prior to this we had this vector had"}, {"start": 1544.24, "end": 1549.0, "text": " like embedding vector V and now after this accumulation across the"}, {"start": 1549.0, "end": 1554.42, "text": " neighborhood so we're somehow modifying these feature vectors we are somehow"}, {"start": 1554.42, "end": 1559.4, "text": " combining them and we get V prime that's the updated representation so now the"}, {"start": 1559.4, "end": 1564.16, "text": " difference here is between get and in general the graph neural networks is"}, {"start": 1564.16, "end": 1568.76, "text": " they have explicit graph structure so compared to that amputees the"}, {"start": 1568.76, "end": 1572.3200000000002, "text": " nonparametric transformers now do the following it is something similar that"}, {"start": 1572.3200000000002, "end": 1577.2, "text": " transformer there with like like with with sentences basically it assumes a"}, {"start": 1577.2, "end": 1581.6000000000001, "text": " fully connected graph so it assumes that all of these are connected with each"}, {"start": 1581.6000000000001, "end": 1585.6000000000001, "text": " other okay I think I've connected everything basically and now it just"}, {"start": 1585.6, "end": 1590.6399999999999, "text": " learns which of these edges have what kind of strength and so it's implicitly"}, {"start": 1590.6399999999999, "end": 1594.36, "text": " inferring the structure of the graph without being said like which edges"}, {"start": 1594.36, "end": 1598.36, "text": " exist on which edges does not exist so it's a more of a soft assignment here"}, {"start": 1598.36, "end": 1602.04, "text": " compared to graph neural network so that's the first connection second"}, {"start": 1602.04, "end": 1605.6, "text": " things second thing I want to mention is that like graph attention network is"}, {"start": 1605.6, "end": 1609.8799999999999, "text": " invariant to the data point ordering so it's not active variant it's invariant"}, {"start": 1609.8799999999999, "end": 1615.1599999999999, "text": " that means no matter how you have these node feature vectors in your table so"}, {"start": 1615.16, "end": 1619.48, "text": " basically let's say this is one this is two this is three you'll represent your"}, {"start": 1619.48, "end": 1623.8400000000001, "text": " graph as certain table where again here we have data points and here you have"}, {"start": 1623.8400000000001, "end": 1628.8400000000001, "text": " number of attributes so now no matter if you if you shuffle these one two three"}, {"start": 1628.8400000000001, "end": 1633.64, "text": " like in whatever permutation you'll still have the same V prime feature"}, {"start": 1633.64, "end": 1640.24, "text": " vector here because what graph attention network does is it just sums sums up"}, {"start": 1640.24, "end": 1647.76, "text": " these these these values and basically some enables us to have these this"}, {"start": 1647.76, "end": 1652.64, "text": " invariance property right because some doesn't care about the permutation of"}, {"start": 1652.64, "end": 1657.08, "text": " your input so on the other hand let me take a convolutional neural network"}, {"start": 1657.08, "end": 1660.4, "text": " just to explain the active variance part I think you'll find it useful"}, {"start": 1660.4, "end": 1665.28, "text": " basically let's assume we have an image here and in this image we have certain"}, {"start": 1665.28, "end": 1670.36, "text": " like an image of a square okay and now if we pass that to a CNN and let's"}, {"start": 1670.36, "end": 1674.24, "text": " assume we're in the first layer of the convolutional neural network and that the"}, {"start": 1674.24, "end": 1680.52, "text": " the first kernel the first filter learns to just figure out the vertical edges in"}, {"start": 1680.52, "end": 1684.6399999999999, "text": " in your in your image so what the image will look like is something like this"}, {"start": 1684.6399999999999, "end": 1689.36, "text": " we'll have something like this and we'll have like two vertical edges here okay so"}, {"start": 1689.36, "end": 1693.56, "text": " the active variance means the following if we now take this square and we"}, {"start": 1693.56, "end": 1701.56, "text": " translate it here maybe here now what will happen in the output is this"}, {"start": 1701.56, "end": 1708.0, "text": " response will just shift by the same amount here so as you can see it's"}, {"start": 1708.0, "end": 1712.44, "text": " equivalent it's varying with input it's not invariant to the position of the"}, {"start": 1712.44, "end": 1718.6799999999998, "text": " square it's actually varying if that's even a verb with the input okay so the"}, {"start": 1718.68, "end": 1724.04, "text": " invariant part income nets usually comes at the like at the classification head"}, {"start": 1724.04, "end": 1728.52, "text": " part because there you really don't care you just care that you have a square in"}, {"start": 1728.52, "end": 1731.8, "text": " the image if you're trying to classify it you don't care where it is so it's"}, {"start": 1731.8, "end": 1736.3200000000002, "text": " actually desirable property to have invariance at the latest stages of the"}, {"start": 1736.3200000000002, "end": 1740.3600000000001, "text": " CNNs but like first stages have active variance so that was just a short"}, {"start": 1740.3600000000001, "end": 1745.72, "text": " summary of what's a covariance what's invariance and so basically now that we"}, {"start": 1745.72, "end": 1750.3600000000001, "text": " know that we can say two things first things first is that amputees are"}, {"start": 1750.3600000000001, "end": 1753.92, "text": " active variant second thing is that they are implicitly learning the graph"}, {"start": 1753.92, "end": 1759.0, "text": " structure over the data set and that's really cool but unfortunately it's a"}, {"start": 1759.0, "end": 1763.1200000000001, "text": " kind of computationally expensive and yeah they say here we'll leave further"}, {"start": 1763.1200000000001, "end": 1766.32, "text": " exploration of the close connection between amputees and graph neural"}, {"start": 1766.32, "end": 1769.64, "text": " networks to future work I just briefly mentioned a couple of obvious"}, {"start": 1769.64, "end": 1773.92, "text": " properties I guess some of the more knowledgeable people from the graph ML"}, {"start": 1773.92, "end": 1777.92, "text": " community could write down comments of what interesting other properties this"}, {"start": 1777.92, "end": 1783.88, "text": " thing has with graph neural networks okay that was it now let me jump to"}, {"start": 1783.88, "end": 1788.16, "text": " experiments they have some really nice experiments okay the first experiment"}, {"start": 1788.16, "end": 1793.72, "text": " they do is they they basically test this on a tabular data set a collection of"}, {"start": 1793.72, "end": 1798.44, "text": " tabular data sets ten of them to be precise and they say it here amputees"}, {"start": 1798.44, "end": 1803.52, "text": " perform competitively on established benchmarks tabular data is"}, {"start": 1803.52, "end": 1808.04, "text": " ubiquitous in real-world machine learning but notoriously challenging for"}, {"start": 1808.04, "end": 1813.08, "text": " general purpose deep neural networks which consistently underperform boosting"}, {"start": 1813.08, "end": 1819.0, "text": " methods and are rarely used in practice so again short note boosting methods"}, {"start": 1819.0, "end": 1823.28, "text": " they are super simple you basically just take take a weak learner which means"}, {"start": 1823.28, "end": 1827.28, "text": " it's a some some really weak model that can be a bit better than random guessing"}, {"start": 1827.28, "end": 1831.96, "text": " and they combine these in a clever way for example majority vote or something"}, {"start": 1831.96, "end": 1836.44, "text": " to get something called strong learner which actually has some nice accuracy"}, {"start": 1836.44, "end": 1841.64, "text": " let's say if we're doing classification so now there was a bunch of like"}, {"start": 1841.64, "end": 1846.02, "text": " different work that came after the original boosting method so there is"}, {"start": 1846.02, "end": 1850.0, "text": " first thing is this add a boost so there's just adaptive boosting where once"}, {"start": 1850.0, "end": 1856.04, "text": " you create the first week a classifier let's say you basically use the data"}, {"start": 1856.04, "end": 1861.16, "text": " points is performing poor on to feed those into the next week learner which"}, {"start": 1861.16, "end": 1866.6200000000001, "text": " will learn to perform much better on the points where this first learner was"}, {"start": 1866.6200000000001, "end": 1870.72, "text": " having difficulties with so that's the add a boost and we had gradient boosting"}, {"start": 1870.72, "end": 1874.18, "text": " and then there is a bunch of variations on gradient boosting so here you can see"}, {"start": 1874.18, "end": 1878.88, "text": " XGBoost which is just a performant like implementation of that gradient"}, {"start": 1878.88, "end": 1883.88, "text": " boosting method we have cat boost also gradient boosting implementation light"}, {"start": 1883.88, "end": 1889.92, "text": " gradient boosting method again gradient boosting and yeah so you get the point"}, {"start": 1889.92, "end": 1893.0800000000002, "text": " these boosting methods are pretty popular on tabular data you'll see them"}, {"start": 1893.0800000000002, "end": 1898.52, "text": " all around kegel and now let's see the results they got and they're pretty"}, {"start": 1898.52, "end": 1904.1200000000001, "text": " spectacular if you take a look at this binary classification tasks so I think"}, {"start": 1904.1200000000001, "end": 1907.6000000000001, "text": " there was four of these you can see MPT is the first in this table it's better"}, {"start": 1907.6000000000001, "end": 1913.16, "text": " than then cat boost and light GBM actually boosts altogether it's also the"}, {"start": 1913.16, "end": 1917.72, "text": " first on this table where they're doing multi-class classification and MPT is"}, {"start": 1917.72, "end": 1922.56, "text": " again the best method it's got a bit less variance compared to XGBoost they're"}, {"start": 1922.56, "end": 1928.48, "text": " otherwise tied here and finally we have regression tasks where MPT was a bit"}, {"start": 1928.48, "end": 1932.8, "text": " worse so you can see it's tied with XGBoost it has a little bit bigger"}, {"start": 1932.8, "end": 1939.52, "text": " variance and cat boost is kind of really really good for this for this task but"}, {"start": 1939.52, "end": 1946.0, "text": " nonetheless they mentioned it here so in addition to its strong rank-wise"}, {"start": 1946.0, "end": 1952.08, "text": " performance MPT achieves best performance on four out of ten benchmark"}, {"start": 1952.08, "end": 1956.76, "text": " datasets more than any other method so they definitely proven that this neural"}, {"start": 1956.76, "end": 1961.96, "text": " network can be used for tabular data and give some nice results by the way if"}, {"start": 1961.96, "end": 1966.44, "text": " you're asking yourself maybe it's not fair like they're they're probably used"}, {"start": 1966.44, "end": 1971.0, "text": " much more computation to train the MPT that's actually not true because for"}, {"start": 1971.0, "end": 1974.88, "text": " just figuring out the best hyperparameters for those boosting methods"}, {"start": 1974.88, "end": 1979.16, "text": " they actually wrote something some somewhere in the paper that the"}, {"start": 1979.16, "end": 1985.96, "text": " computation necessary was pretty much on pair so as so this is supporting our"}, {"start": 1985.96, "end": 1990.1200000000001, "text": " hypothesis that attention between data points is a useful architectural"}, {"start": 1990.1200000000001, "end": 1997.5200000000002, "text": " inductive bias for prediction and aside from these tabular datasets they also"}, {"start": 1997.5200000000002, "end": 2004.2800000000002, "text": " tested MPT on the image kind of benchmarks like I think MNIST and Cypher-10"}, {"start": 2004.28, "end": 2008.32, "text": " and they said here similar to previous work on transformers for computer vision"}, {"start": 2008.32, "end": 2013.44, "text": " we would expect pre-training a millions of images to significantly boost MPT's"}, {"start": 2013.44, "end": 2017.8799999999999, "text": " performance and if you watch my previous video you now know that vision"}, {"start": 2017.8799999999999, "end": 2022.72, "text": " transformers and MLP mixers don't need pre-training at all if you use this"}, {"start": 2022.72, "end": 2028.24, "text": " thing called sharpness aware minimization objective and again I'll"}, {"start": 2028.24, "end": 2032.04, "text": " link the video somewhere here you can check it out basically if we have if you"}, {"start": 2032.04, "end": 2038.0, "text": " somehow find a way to to smoothen out the lost landscape you'll notice that"}, {"start": 2038.0, "end": 2041.92, "text": " you don't need extensive pre-training anymore and you can achieve results that"}, {"start": 2041.92, "end": 2046.76, "text": " outperform resonant baselines and that's it in a nutshell so I guess the authors"}, {"start": 2046.76, "end": 2050.88, "text": " of the paper either either didn't have the time to rewrite this or they still"}, {"start": 2050.88, "end": 2056.2, "text": " haven't read that paper which recently came out a couple of days ago okay"}, {"start": 2056.2, "end": 2062.64, "text": " that's the first part now let's see some some some cool things this diagram here"}, {"start": 2062.64, "end": 2071.68, "text": " let me just introduce it basically they make this semi synthetic the problem and"}, {"start": 2071.68, "end": 2076.12, "text": " they say it here for each batch we input the original data with masked target"}, {"start": 2076.12, "end": 2080.9199999999996, "text": " values as well as a copy of the original data where all target values have been"}, {"start": 2080.92, "end": 2086.96, "text": " revealed I know masking is applied at this time we input novel semi synthetic"}, {"start": 2086.96, "end": 2092.16, "text": " test data to ensure that MPT has learned the correct relational mechanism as"}, {"start": 2092.16, "end": 2098.6800000000003, "text": " opposed to just memorizing the target values okay so they they have a nice"}, {"start": 2098.6800000000003, "end": 2103.7200000000003, "text": " nice chart here where you can see the what's what's the input so this is the"}, {"start": 2103.7200000000003, "end": 2107.92, "text": " input we have some data points and we have the target values there are hidden"}, {"start": 2107.92, "end": 2114.04, "text": " denoted by these question marks and we have the actual correct values here so"}, {"start": 2114.04, "end": 2119.32, "text": " if MPT smart even though we're gonna just back prop on these masked values is"}, {"start": 2119.32, "end": 2124.6800000000003, "text": " going to learn how to attend how to find this similar vector so this is basically"}, {"start": 2124.6800000000003, "end": 2129.28, "text": " the same vector and then just copy this value here instead of learning to"}, {"start": 2129.28, "end": 2134.28, "text": " somehow map and use the attributes is going to learn the relational mechanism"}, {"start": 2134.28, "end": 2138.8, "text": " and why do we want that well because it's going to generalize much better if"}, {"start": 2138.8, "end": 2143.0800000000004, "text": " it learns how to just attend to the same vector and just copy the value that's"}, {"start": 2143.0800000000004, "end": 2148.6400000000003, "text": " the whole point of relational reasoning so as you can see here they draw some"}, {"start": 2148.6400000000003, "end": 2152.8, "text": " attention maps so this green data point so that's this one the first one it"}, {"start": 2152.8, "end": 2157.52, "text": " attends to this one so that's this data point here so that means it learned"}, {"start": 2157.52, "end": 2163.0800000000004, "text": " completely and you can see attention here is one all around you have zeros if"}, {"start": 2163.08, "end": 2168.12, "text": " you take a look at this heat map so that means the whole attention of this data"}, {"start": 2168.12, "end": 2172.6, "text": " point of the first data point is on this one and that's exactly what we wanted"}, {"start": 2172.6, "end": 2177.24, "text": " and you can see the pattern going for all of these data points here and here"}, {"start": 2177.24, "end": 2181.68, "text": " this part you can ignore they didn't enforce this they did write something"}, {"start": 2181.68, "end": 2186.04, "text": " about this in the appendix why these data points are attending to themselves"}, {"start": 2186.04, "end": 2190.36, "text": " so that's something may be interesting taking a look into but yeah basically"}, {"start": 2190.36, "end": 2195.96, "text": " you can see the results are really cool Pearson correlation is 99.9 that means"}, {"start": 2195.96, "end": 2200.2400000000002, "text": " basically a fancy way of saying that it basically learns how to copy over these"}, {"start": 2200.2400000000002, "end": 2204.96, "text": " values or that's a hypothesis for now but like it's it's it's basically"}, {"start": 2204.96, "end": 2209.0, "text": " outputting perfect values as if it was copying them and we'll soon see that"}, {"start": 2209.0, "end": 2214.4, "text": " it's actually copying them so for this second row let me just go back down here"}, {"start": 2214.4, "end": 2218.92, "text": " okay so additionally we perform an interventional experiment to"}, {"start": 2218.92, "end": 2222.64, "text": " investigate the extent to which amputees have actually learned the causal"}, {"start": 2222.64, "end": 2227.44, "text": " mechanism underlying the lookup task and if you see these two keywords and if"}, {"start": 2227.44, "end": 2231.56, "text": " you're familiar with Judea Pearl's work you'll notice that this is basically"}, {"start": 2231.56, "end": 2237.48, "text": " rank two experiments so we have this thing called letter of causation the"}, {"start": 2237.48, "end": 2243.0, "text": " first letter is just observing data the second letter would be having acting in"}, {"start": 2243.0, "end": 2247.2000000000003, "text": " the environment or doing these these interventional kinds of experiments"}, {"start": 2247.2, "end": 2250.8799999999997, "text": " where you're setting something to certain value and then observing and"}, {"start": 2250.8799999999997, "end": 2256.52, "text": " finally we have a counterfactual level where we are imagining like different"}, {"start": 2256.52, "end": 2260.8399999999997, "text": " worlds and that sounds super abstract for this context but yeah just just I"}, {"start": 2260.8399999999997, "end": 2265.8799999999997, "text": " wanted to just associate this sentence here with Judea Pearl's work so if you"}, {"start": 2265.8799999999997, "end": 2270.08, "text": " want to know more about it you can check some of my previous LinkedIn posts or"}, {"start": 2270.08, "end": 2275.6, "text": " you can read the book of why from from Judea Pearl as a nice introduction to"}, {"start": 2275.6, "end": 2280.36, "text": " his work so the model is not confronted with target values associated with"}, {"start": 2280.36, "end": 2285.2799999999997, "text": " features that are highly unlikely under the training data and let's see why that"}, {"start": 2285.2799999999997, "end": 2291.04, "text": " is so basically what I do as you can see so it's the same experiment as the above"}, {"start": 2291.04, "end": 2297.8399999999997, "text": " but this time they take this target value and instead of basically keeping"}, {"start": 2297.8399999999997, "end": 2303.16, "text": " it the way it was in the original data set they tweak it they take that maybe"}, {"start": 2303.16, "end": 2310.24, "text": " there was 1.3 and they're going to kind of mess it up and put 7.5 here and that"}, {"start": 2310.24, "end": 2315.2799999999997, "text": " combination 7.5 with similar attributes have never been seen in the training"}, {"start": 2315.2799999999997, "end": 2320.7599999999998, "text": " data set so now if the model learned how to just kind of use the attributes and"}, {"start": 2320.7599999999998, "end": 2325.0, "text": " figure out the target value is gonna fail here and on the other hand if you'd"}, {"start": 2325.0, "end": 2328.8399999999997, "text": " learn how to do what I previously mentioned and that's just to attend to"}, {"start": 2328.84, "end": 2333.1600000000003, "text": " the vector and copy that value it's gonna have nice performance and as we"}, {"start": 2333.1600000000003, "end": 2338.6000000000004, "text": " can see here the Pearson correlation is again great that means it learned how to"}, {"start": 2338.6000000000004, "end": 2344.96, "text": " no matter these interventions it learned how to predict the exact value from from"}, {"start": 2344.96, "end": 2349.56, "text": " these slots and that means it learned the correct relational mechanism that's"}, {"start": 2349.56, "end": 2354.48, "text": " the point of these experiments and so we now confidently conclude that the MPTs"}, {"start": 2354.48, "end": 2358.76, "text": " robustly learn the causal data generating mechanism underlying the"}, {"start": 2358.76, "end": 2363.76, "text": " semi-synthetic data set this requires MPTs to learn a non-trivial sequence of"}, {"start": 2363.76, "end": 2368.28, "text": " computational steps they must learn to match rows based on similarity of"}, {"start": 2368.28, "end": 2373.76, "text": " relevant features to look up the target value of the duplicated data point and"}, {"start": 2373.76, "end": 2378.08, "text": " to copy that value into the target of the mask data point so that's what I"}, {"start": 2378.08, "end": 2383.2400000000002, "text": " already explained and yeah hopefully now it makes more sense okay next"}, {"start": 2383.24, "end": 2388.4399999999996, "text": " experiment they did was to corrupt the data points and see whether it's failing"}, {"start": 2388.4399999999996, "end": 2393.0, "text": " because it learned how to actually rely on those other data points so what I do"}, {"start": 2393.0, "end": 2396.7999999999997, "text": " they said here so this completely corrupts information from all data"}, {"start": 2396.7999999999997, "end": 2401.16, "text": " points except except the one for which we evaluate hence a model that relies"}, {"start": 2401.16, "end": 2404.4799999999996, "text": " meaningfully on attention between data points will show deteriorating"}, {"start": 2404.4799999999996, "end": 2411.56, "text": " performance so let me just explain you what they actually did and they did the"}, {"start": 2411.56, "end": 2418.16, "text": " following thing so we have a data set here and let's say we have a bunch of"}, {"start": 2418.16, "end": 2422.6, "text": " data points and let's say we want to predict this value here okay so this is"}, {"start": 2422.6, "end": 2427.4, "text": " the data point for which we want to figure out predict the class what I do"}, {"start": 2427.4, "end": 2432.04, "text": " is they just shuffle all of these attributes of all of the others so all"}, {"start": 2432.04, "end": 2437.08, "text": " of these other data points are going to get shuffled and that means this feature"}, {"start": 2437.08, "end": 2443.2, "text": " vector can't be matched now with other feature vectors in them like using the"}, {"start": 2443.2, "end": 2446.6, "text": " we can't use the mechanism we just described it's using and that's just"}, {"start": 2446.6, "end": 2450.52, "text": " figuring out matching the vector finding maybe I don't know maybe it learned to do"}, {"start": 2450.52, "end": 2454.7599999999998, "text": " cosine similarity now I can't do that because we shuffle the attributes and"}, {"start": 2454.7599999999998, "end": 2459.48, "text": " what I show is that performance deteriorates which is a further proof"}, {"start": 2459.48, "end": 2464.52, "text": " that it learned the correct relational mechanism it appears that when MPTs do"}, {"start": 2464.52, "end": 2468.2, "text": " not find it advantageous to rely on attention between data points during"}, {"start": 2468.2, "end": 2472.12, "text": " training they can learn to completely ignore the other inputs essentially"}, {"start": 2472.12, "end": 2476.2, "text": " collapsing into a standard parametric model this supports our earlier claims"}, {"start": 2476.2, "end": 2481.0, "text": " that amputees can learn end-to-end from data the extent to which they rely on"}, {"start": 2481.0, "end": 2484.4, "text": " other data points for prediction so that's a fancy way of saying that"}, {"start": 2484.4, "end": 2489.6, "text": " basically sometimes if we have a data set again so sometimes it's actually"}, {"start": 2489.6, "end": 2495.44, "text": " just relying on this particular row on a single data point to predict this value"}, {"start": 2495.44, "end": 2498.8399999999997, "text": " here because for that task you figure out it doesn't actually need to use"}, {"start": 2498.8399999999997, "end": 2504.7999999999997, "text": " other data points and for those data sets the the performance they show does"}, {"start": 2504.7999999999997, "end": 2509.64, "text": " not deteriorate which is again a further proof about proof informal proof that"}, {"start": 2509.64, "end": 2513.92, "text": " basically it learned the correct mechanism here they just have a table a"}, {"start": 2513.92, "end": 2519.48, "text": " bunch of different data sets you can see regressions here so yeah on certain"}, {"start": 2519.48, "end": 2524.08, "text": " data sets as I just mentioned it's not deteriorating that much because it"}, {"start": 2524.08, "end": 2529.72, "text": " doesn't rely on other data points that much okay having said that let me see if"}, {"start": 2529.72, "end": 2534.64, "text": " we have something else interesting and yeah this is the last part that's super"}, {"start": 2534.64, "end": 2539.72, "text": " interesting and they now try to figure out how the model finds those similar"}, {"start": 2539.72, "end": 2544.76, "text": " vectors so and what I do is we sort the input data with respect to their feature"}, {"start": 2544.76, "end": 2548.58, "text": " space it's actually input space as I saw in the app appendix but maybe the"}, {"start": 2548.58, "end": 2553.56, "text": " authors can correct me here distance such that the similar data points are"}, {"start": 2553.56, "end": 2561.12, "text": " now close to each other and so let me explain what that means basically that"}, {"start": 2561.12, "end": 2566.7599999999998, "text": " means you have the original tabular data set you have your data points what I do"}, {"start": 2566.7599999999998, "end": 2572.3199999999997, "text": " is they find maybe using L2 metric define the closest vectors here and"}, {"start": 2572.3199999999997, "end": 2577.48, "text": " they're just going to sort them so now we have a vector here and a vector here"}, {"start": 2577.48, "end": 2582.32, "text": " and these two have really small L2 distance and now we have you know vector"}, {"start": 2582.32, "end": 2586.92, "text": " here and these two have really small L2 distance which that means that basically"}, {"start": 2586.92, "end": 2592.08, "text": " these two also have maybe a bit bigger L2 distance okay so you get the point"}, {"start": 2592.08, "end": 2598.96, "text": " basically we're slowly adding stacking vectors which are further away looking"}, {"start": 2598.96, "end": 2604.08, "text": " at the L2 distance and when we do that and we apply NPT and when we look at"}, {"start": 2604.08, "end": 2610.2, "text": " the certain like attention patterns from a certain like head we get this pattern"}, {"start": 2610.2, "end": 2615.16, "text": " and this means that is basically attending to the vectors which are"}, {"start": 2615.16, "end": 2620.6, "text": " basically close in the input space which means it's attending to similar vectors"}, {"start": 2620.6, "end": 2624.96, "text": " according to this L2 distance and if you don't know how to parse this diagram"}, {"start": 2624.96, "end": 2632.7599999999998, "text": " let me help you basically if we have if this is row I this row is going to"}, {"start": 2632.76, "end": 2637.6800000000003, "text": " attend and assume we have a sort of table so this table is sorted according"}, {"start": 2637.6800000000003, "end": 2642.96, "text": " to L2 so let's say this is I this is vector I and it's somewhere in the"}, {"start": 2642.96, "end": 2647.36, "text": " middle of the table and that vector is going to attend to its proximity so it's"}, {"start": 2647.36, "end": 2651.0, "text": " going to attend to these vectors really strongly and then it's going to drop off"}, {"start": 2651.0, "end": 2654.88, "text": " and then it's going to have really weak attendance to maybe this vector here so"}, {"start": 2654.88, "end": 2660.1600000000003, "text": " it's going to tend to these vectors really strongly and the strength will"}, {"start": 2660.16, "end": 2666.16, "text": " kind of fade away going downwards or upwards here and that's exactly what you"}, {"start": 2666.16, "end": 2671.0, "text": " see in this attention map basically if this is vector I and this is just n"}, {"start": 2671.0, "end": 2675.12, "text": " number of data points this is just n number of data points this means that"}, {"start": 2675.12, "end": 2679.92, "text": " this vector as you can see attends to itself really strongly and to its"}, {"start": 2679.92, "end": 2684.56, "text": " neighborhood but then it deteriorates because you can see that the dark blue"}, {"start": 2684.56, "end": 2689.44, "text": " means the attention coefficients are really really really weak there okay"}, {"start": 2689.44, "end": 2694.16, "text": " that's the whole point and yeah further they had some additional experiment"}, {"start": 2694.16, "end": 2697.96, "text": " where they're doing similar thing but the conclusion was the same basically"}, {"start": 2697.96, "end": 2702.56, "text": " MPT learns how to attend to those vectors which have which are really"}, {"start": 2702.56, "end": 2709.32, "text": " close in the input space looking at the L2 distance okay that's pretty much it"}, {"start": 2709.32, "end": 2713.7200000000003, "text": " they dimension some problems like these scaling limitations which I have already"}, {"start": 2713.72, "end": 2721.52, "text": " flagged and that's pretty much it so we saw that MPT's learn how to do"}, {"start": 2721.52, "end": 2726.24, "text": " relational reasoning and because they learn the correct mechanism on that"}, {"start": 2726.24, "end": 2730.3599999999997, "text": " simple semi-synthetic task it can generalize much better than if they"}, {"start": 2730.3599999999997, "end": 2735.12, "text": " actually learn how to deduce the value just looking at the attributes and"}, {"start": 2735.12, "end": 2738.8399999999997, "text": " instead they learn how to just copy paste the values from the corresponding"}, {"start": 2738.8399999999997, "end": 2742.72, "text": " call of Rose and yeah hopefully this was useful if you found this video useful"}, {"start": 2742.72, "end": 2748.08, "text": " share it out and subscribe and until next time bye bye"}]