How to get started with Graph ML? (Blog walkthrough)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I walk you through my blog on getting started with Graph ML.
I talk about research, learning, cool Graph ML apps, resources to get you started, my GAT project, and beyond! (till infinity)

You'll learn about:
✔️ Various research tips
✔️ How to get started with Graph ML

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Medium blog: https://gordicaleksa.medium.com/how-to-get-started-with-graph-machine-learning-afa53f6f963a
✅ GAT project: https://github.com/gordicaleksa/pytorch-GAT
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 Research/learning challenges
03:05 What is Graph ML? We're all graphs
05:05 Cool Graph ML applications
09:35 Fake news and fundamental science
13:20 Halicin a potent antibiotic discovered by a GNN
15:45 Contrasting Graph ML with CV and NLP
19:15 Resources - graph embedding methods
21:10 Graph Neural Networks
23:20 Top to bottom approach - high level resources
26:35 Spatial methods
30:40 Simple baselines sometimes work great!
32:25 Parallel with CNNs
33:40 GNN expressivity
36:35 Dynamic graphs
39:15 Unsupervised graph learning and geometric DL
40:55 Datasets/benchmarks and newsletter
44:15 GAT project
44:50 Related research subfields

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#graphml #graphs #deeplearning
 What's up folks? So in this video I'm going to tell you more about how to get started with GraphML. So I'm wrapping up my exploration of Graph Machine Learning and I wrote this blog on how to get started. So I thought of just creating a video and maybe telling you more about my opinions and some of the stuff that I've already written in the blog and giving you some complimentary ideas. So yeah, basically stay tuned and hopefully you'll learn something new. Okay so the first thing I want to talk about is this. So research can be really tough and just be aware that everybody, every single like top-notch researcher you know of or engineer or successful business person, whatever, has hard days and it's not always easy. So I'm talking from my own experience. So it's not always mentally easy to stick with the plan and execute and do the research, read the papers, code up that project. So just be aware of it that you're not alone in this and you should never give up. That's my only advice I have to give you. Never freaking give up. Okay? Secondly, don't get demotivated by the people who are, who seem, who appear to be super successful in something. So the thing is most of them are really specialized. So I like to think from about this from like this linear algebra perspective. So maybe nerdy but there is a method to this madness. Basically I want to say the following. So among all of those different dimensions, all of the things you could be doing researching, learning, about, let's take two axes. So one axis would be maybe somebody who is just reading research papers. So some fictive person that's only reading research. On the other hand you have people who are just coding something, who are working on some deep learning libraries, like maybe in the context of GraphML that would be PyTorch Geometric. And most people tend to be somewhere in between. But there are a lot of people who only read research paper and they never implement something from scratch or never work on anything that has to do with engineering or coding. And I want to tell you that they don't know many details even though they appear to understand everything because they curate their own content. They don't know many details of how stuff works. And I'm talking from my personal experience. So until I start coding something up from scratch I didn't, I don't understand thoroughly how much I actually don't know, if that makes sense. So you want to be somewhere in between. If you want to be a really successful researcher, in my opinion, and you want to boost that H index if that's of something of interest to you, you should probably be reading, focusing on reading research papers a lot. But you should also make these, let's call them detours into implementing something and actually playing with the code. On the other hand if you want to be a great engineer and you probably want to focus on code but you still want to research and read, allocate some time to read research papers because otherwise you'll just lose the pace and you'll, the field is progressing very rapidly and you got to allocate some time to do the due diligence. Having said that, let me walk you through this blog. So first GraphML and hopefully you know what GraphML is. You heard about graphs so those are those data structures and GraphML is just a bunch of machine learning models so we learn on top of that graph data. And the first thing I want to advise you here is don't, if you're, especially if you're a perfectionist, you'll tend to be like this. I want to make a, I want to go on a tangent here and maybe invest three months of my time and get some solid background in graph theory, a branch of mathematics. Or maybe get a really good foundations in the network analysis field. And I want to tell you, you don't need to do that. You can jump straight into the field and make a first step and without doing all of those detours. So it's totally fine to start reading this blog and start with high-level resources and that would be your, as a first step to on your journey through Graph Machine Learning. Sometimes you may want to take a step back or maybe better said, make a side step and invest maybe three months of your time into building up more solid mathematical foundations. And me personally, last year in 2020, I did maybe, I invested maybe around three, four months into reading this book, Mathematics for Machine Learning. And it's a bit complex if you're intermediate or even beginner. But it meant a lot for me and it helped me better understand the stuff that I later learned. So the strategy of doing side steps and investing into your some foundations is also a good idea sometimes. So that's GraphML. You have machine learning, you have machine learning models that learn on graph data. Instead of images or text like in computer vision or NLP, you're learning from graphs. And now before we start, before I show you the resources, let's just, let me just walk you through some cool applications. And one more thing to note, you'll hear a lot about graph neural networks here and for the sake of, for now just think of them as neural networks that work with graphs and somehow can process those graphs and do the classification or regression on nodes or edges or even the complete graphs. So just have them as a black box on top of your mind, okay? So first things first, GraphML is such an exciting topic because the applications you'll see here are so much different from those you'll see in the computer vision or the NLP worlds. So there in computer vision you'll see, you'll be seeing algorithms that manipulate images or videos and in NLP you'll be seeing algorithms that manipulate text and those are all pretty homogeneous. So even though you can do, I don't know, facial landmark detection, pose estimation, bunch of things, segmentation, regressing 3D models, bunch of things in computer vision, they are all still kind of related. And then when you start doing GraphML you understand like all the plethora, like the vast diversity of different applications you can do with graphs. So two cool applications I start, I want to start with our recommendation systems and if you're not aware already, like in my opinion, recommendation systems are the single most important and influential piece of AI currently in 2021. So you interact with with recommendation systems all the time, YouTube recommendation, Facebook, Twitter, Pinterest, whatever, just name it, and you have recommendation systems influencing your decisions. So before I started making YouTube videos I wasn't fully aware of it and now I'm getting a thorough understanding of how how influential recommender systems are. So basically as an example I've seen many videos which are really top-notch, high quality, that have 5k views and not because the presenter was anything less eloquent or the content was worse, no. It was just because he or she came on later and so he or she is not on the on the front page, like among the first queries. And then you'll see some videos which were made maybe 2015, 16, 17 and they have amassed a lot of views and so that's what YouTube will recommend you. So what I've learned is I now tend to go past the first five results before I would just open the most viewed videos but now I'm completely aware how recommendation systems can actually bring you inferior recommendations, unfortunately. And yeah, engineers and researchers are working on this and one way is to just kind of use graph neural networks. So because the problem as you can see here with with uBreads for example, you can really nicely model all of these problems as graphs. So here you have a graph between users and food and uBreads has a separate graph between users and restaurants. They still haven't merged the two together even though I think they're currently working on it or maybe they even maybe they recently even deployed it. I'm not aware of it but yeah this is the last news I'm aware of. And so just processing this graph data you can make better recommendations to your users and obviously I won't get into all of the details. I did some videos on Pinsage, that's the recommendation engine at Pinterest and you can check out that one somewhere here in the card. Okay, so you can kind of see how the graph neural network Pinsage improve upon these baselines. Here you can see people carrying some logs and here you can see the visual. If you just take the visual baseline it will bring you images that are like really similar when it comes to appearance because they are also grayscale but you can see the semantics like here the soldiers and here you can see actually the trees and the logging so Pinsage got both the semantics as well as the appearance successfully by leveraging both the graph structure as well as the visual textual information that's available in those graphs on those platforms. So those are some nice applications and aside from from recommendation systems we have these basically fake news detection GNNs and that's so important. The societal benefits of deploying really successful fake news detection algorithms are so important and again you can see here maybe on Twitter you have a single tweet and somebody retweets a tweet that would be a second node and you have the edge and so you can build up the graph and basically these networks can figure out the propagation patterns of those retweets and just by looking at the propagation patterns it was shown in previous research that we can figure out whether the news is actually fake or real and again that's a really naturally model as a graph so GNNs to the rescue. Aside from this app there is one more cool app which is basically figuring out the probability that you will produce certain event depending on your neighborhood and whether they did the same thing so for example to be less abstract maybe the goal is to figure out whether you'll retweet a tweet if some of your neighbors retweeted the same tweet that would be one concrete example and again you have the graph structure and you can leverage to regress these probabilities so various cool applications and what's even more interesting is that graph ML so all of these are still kind of familiar to you but like graph ML goes into these deep fundamental science problems and can naturally solve them so one of them would be this example I gave with the neutrino detection where basically the setup is the following you have a huge huge cube of ice and you have these light sensors inter dispersed in the like distributed throughout this ice and what happens is a neutrino comes comes out from the space and hits the ice and interacts with the ice molecules and in that interaction something that's called charank of light emits and basically those sensors those light sensors can catch those that light and you can basically see certain nodes of that grid like structure of sensors get activated and then you can just deploy the graph neural network on top to figure out whether the neutrino came from some certain events we care about like supernova explosions or whatever so I don't know about you but like this completely blew out my mind because usually you see neural networks applied to images videos text and that's it and here you can see fundamental science and GNN is being used to solve them to solve those problems okay yeah there are many other apps I'll skip some of them like figuring out the molecules that can help beat cancer because you can see you can model the the thing naturally as a graph again you have genes you have proteins all of those interactions and here's some molecules or drugs and you you model that as a graph and you can find interesting information like which molecules can help treat cancer better or prevent cancer then you have drug side effects when you combine multiple drugs you want to be sure it would be beneficial if we could computationally calculate at the possible side effects because the combinatorics is so huge here and it's impractical to do it in a lab setting in vivo we need to do it in in silica or computationally so many cool applications so the last one I'll note here is the one that it was gained a lot of traction last year in 2020 basically we as a society have a huge problem with with anti antibiotic resistant bacteria so the reason that happens is because people are misusing antibiotics they're using that them for viruses they're using them to treat common colds and you shouldn't be using antibiotics in those cases you should be using antibiotic a specific antibiotic if you're sure you have a certain bacteria that this antibiotic can inhibit and because of all of those misuse there is a threat of this bacteria which is resistant to every single antibiotic and that's like the worst nightmare in the like in the science community and in the medicine in general pharmacy and so what this graph neural network did is it was trained on a set of these of these specific molecules and then and was trained to figure out which of these molecules tend to inhibit one specific bacteria namely the asharica coli in this example and then once they deployed that trained graph neural network onto a specific new subset a new data set it was able to filter out hundred candidates that gave a high response a high probability that this molecule will inhibit asharica coli and then they just dedicated that small small subset of molecules to scientists and they were able to find this molecule called helicin which basically can treat which was basically found out to be a super potent antibiotic and extremely successful in inhibiting asharica coli and previously it was just they considered this same molecule for his for its anti-diabetic properties so those are just some of the super exciting applications of graph neural networks and I hope you you found this interesting and and that it motivated you to further investigate the graph ML field now let me just kind of make a brief detour and compare graph ML with computer vision and natural language processing and I personally think it's really important to to try and connect the ideas from various subfields of deep learning because by doing that but we're just creating those connections I think you'll be better to generate much better research ideas and yeah it's just cool to kind of make this connection just gives you some kind of satisfaction as well so basically you can treat computer vision or images and videos in general as a just a special case of graph which is highly regular and we call these grids so you can see here a four by four pixel image can be represented as this specific graph and here we have four neighbors in general you can have up to eight neighbors because right you have the diagonal pixels as well and you have the top left top right bottom left bottom right and that means we can actually deploy graph neural networks to solve computer vision problems and same goes for for NLP because sentence is just basically you can treat sentence either as a fully connected graph or maybe just as a as a tree and yeah but the thing I want to mention here and I mentioned that in the blog as well is don't fall into the trap into thinking that you can you can better solve these problems with graph ML so the thing is certain biases that CNN's have like having weight sharing and localization are super useful and we show that that's much more efficient than using graph ML in some cases though you can use graph neural networks and perform better so when I say that I mean the following transformers are a special case of graph neural networks in a sense so once you have a fully connected graph transformer is basically a graph neural network that operates on a fully connected graph and it was shown in 2020 in October I think the vision transformer that we can use transformers on images to improve upon CNN's if we have enough data so if you have a huge amount of data it is possible maybe to use transformers or graph neural networks to achieve even better results then by using CNN's but there's a saying if you only have hammer everything looks like a nail and it's it's really bad if you if you just have the graph ML context and you don't have if you don't know other fields you're gonna try and and and use graph neural network for everything same goes for deep learning and that's why it's useful to be aware of different fields different methods different techniques because for every specific problem there is a solution that's highly optimized to be used for that specific problem now there is a trade-off between being as general as possible like maybe GPT and on the other hand having a super specialized algorithms so they are better they are more specialized they are more optimized but then you have the overhead of some like developers having to invest a lot of their time to now specialize that's to tune that algorithm for every single problem encounter on the other hand you have this general architectures you just apply them and they work on everything you don't have to invest any more time but they are suboptimal in every single of those problems so that's the trade-off having said all of that let me now walk you through the resources in the way and just kind of advise you how you can structure your learning of graph machine learning okay so if I were you I'd first start with graph embedding methods so these are not graph neural networks these are a specific class of methods which are similar to Vertubec if you're familiar with natural language processing and you know what Vertubec is these are going to be much more familiar and easy to grasp so they are just repurposed Vertubec onto graphs so basically for example you have these these algorithms DeepWalk and No2vec and Planetoid and what Deep2Walk does is the following it just samples random walks from a graph and you can treat those as a sentence so if you treat the graph as a corpus of text and you treat that specific random walk as as a sample sentence it's basically the same thing as Vertubec you just want to make sure that the neighboring words have similar embeddings so if they tend to be close on those in those sentences you want to make the embeddings really close and if they're if they never appear in the same sentence if some words never appear in the same sentence you want to make their embeddings really different so that's the main idea of all of these methods so take some time to kind of understand these because all of these graph embedding methods can be used for exciting graph problems like doing regression on top of those embeddings or classification both on node wise level edgewise and graph wise level so I should you should probably start with these and that's gonna be it's gonna be a nice first step okay in a way it's orthogonal to the graph neural networks that I'm going to tell you about more in a second so graph neural networks when it comes to graph neural networks there are two main branches and one is the spectral methods and the second one is spatial or message passing methods and my advice here is to take some time and investigate spectral methods because the terminology and references will get get propagated into the spatial methods and you'll be seeing some terminology from the spectral methods so it's good to kind of understand at least some basics about these so having said that if you're not familiar if you don't have a strong base in like fundamentals in linear algebra if you don't understand what eigen space is if you don't understand what eigen vector and eigenvalue is if you don't have some understanding of the Fourier analysis or signal processing this is going to be tough like you should probably just read some high-level blogs and try to kind of get familiar with the words and terminology and don't kind of pull yourself down just because you don't understand everything because yeah as I told you you'll need some time to understand it especially if you don't have the fundamentals but having said that just go through it get some feeling because that's gonna be super useful for you believe me so the reason I tell you not to worry about them too much is because they are computationally very expensive because how they work you have something called graph ML so they're just a huge matrix and what you want to do is you want to calculate those eigenvectors and that operation is very expensive so and not only that it's that also these these methods are key and generalized to different graph structures so they they have to be applied in a transductive manner and can be applied to new graphs which are smaller bigger because they inherently are transductive so that's the second pin point of these algorithms but still I told you take some time and learn about them okay in my opinion again in this video is full of opinions you want to start with as always with a top-to-bottom approach you want to start with some high-level videos high-level blocks that will get you exposed to the terminology and new ideas new concepts without you having to understand every single detail so you're just trying to build up that skeleton of knowledge and you should probably start doing that by by watching videos and reading through some medium and towards data science blocks so those are usually really nicely curated high quality blogs and videos and they'll get you exposed to the essential concepts having said that you'll find some of the links here and this video from the from the New York University they are they're kind of publishing all of the lectures they hold there so that's super super super useful for the community and yeah you should start with those I mentioned the Fourier analysis and stuff so if you don't know about if you haven't watched this one even if you know everything about Fourier I strongly suggest you go ahead and watch Grant Sanderson's video on Fourier so when I say Grant Sanderson's he's better known as the three blue one brown channel and he's got some a couple of videos on Fourier which are so nicely visualized and if you don't have the background from Fourier analysis this is probably going to be enough like a high-level understanding of what Fourier is basically decomposing your signal into a sum of differently scaled frequencies and it's gonna be a really good step for you to understand spectrum methods if you don't have any background whatsoever so do watch that one I've linked up to important papers here this one is the most important one spectral networks and locally connected networks and graphs and Chabnets just kind of localized that method local when I say localized I mean you just want to update a single node you'll just use the k-hop not neighborhood instead of using every single node in the graph which can be huge like in some examples when you're having maybe social level graphs you want to make those algorithms localized and the second thing aside from localization Chabnets made it also more efficient to calculate these representations by just leveraging the laplacian graph matrices and not and not calculating the eigenvectors I previously mentioned again some higher-level blogs just go through those research resources you should probably literally sequentially be learning about these resources I've linked you should probably read the graph embedding methods after you do the high-level blogs and videos but whatever feel free to experiment there but you should roughly be following the flow of the blog and as the blog progresses we go into more research and more niche topics like dynamics dynamic graphs and like graph expressivity etc okay having said that let me jump to spatial methods and these are the methods you'll be seeing you'll mostly be seeing so those are networks such as GCN graph convolutional network networks such as get graph attention network then you have graph sage pin sage many different spatial methods and on the high level they all work really similarly so basically you do the following thing so you have a node and you have the neighborhood and what you do is you transform the neighborhood feature vector somehow maybe doing some linear projection or MLP whatever you kind of is you somehow aggregate them maybe doing mean of those feature vectors or some more max or whatever and then you combine the central node its representation with that aggregated transform representation again somehow and that's your new representation and that's the general framework of message passing networks and all of these can be considered as special case of that framework I just explained you I covered many of these spatial methods on my youtube channel so check them out graph attention networks graph convolutional networks pin sage and graph sage so those are the two that were deployed in the recommendation system I previously mentioned at the beginning of the video message passing a neural network is the one that introduced this general framework general notion and understanding that all of these methods are just special case of something and they're doing something fairly similar to each other two more papers you should probably check out is the gated graph neural network which kinda popularized the usage of graph neural networks again because they were initially the similar methods were proposed already I think in 2004 2009 period time frame and this paper kind of popularized GNN's again and it's it's a nice paper it's it's really really general because it also includes the edge features it uses the group the LSTM for the aggregation I just mentioned and so it's a it's a it's a nice read aside from that paper learning convolutional neural networks for graphs is a cool paper because I didn't mention this but like the spectral and spatial methods what they try to do is generalize the concept of convolution that works so nicely in the computer vision world so with CNN's on two graphs and that turned out to be quite a difficult challenging problem in a way and so there were different approaches one was a spectral one one was a spatial one which is basically approximation of the spectral methods so it's not as mathematically rigorous as the spectral methods but a as I already mentioned in practice they work really nice and this paper here just introduced another idea and that was to basically copy the concept from the CNN's more literally than mathematically if that makes sense so basically what they try to do is introduce the concept of node ordering into into their method because the main problem with graphs and the reasons you can't you just can't use a fixed kernel like using the CNN is first of all you have the varying number of nodes sometimes a node will have only a single neighbor sometimes it will have thousand neighbors and you obviously can't use a nine by nine kernel or three by three kernel as it's commonly used in CNN's so that's the first problem the second problem you don't have any concept any any notion of ordering so like in CNN's you have top left pixel you have top right pixel in GNN's you don't have that notion so what they try to do is they try to linearize a neighborhood by doing something called graph labeling and then apply this the kernel that's always fixed onto those order linearized neighborhoods and so it's a nice read the idea itself didn't get that much traction but it's it's an awesome read and so do check it out okay I'm wrapping now with message passing networks with this one simplifying graph convolutional networks paper what they figured out is there are some graph classes like homophilic graphs where you just you can use just much simpler baselines the graph neural networks and so this paper just showed that so homophilic graphs are those graphs where the node where the nodes that are connected have similar label so that that's a common thing you you have in social network types of graphs because usually people had to have similar opinions tend to cluster together and those types of graphs are called homophilic and basically what this paper showed is that you can use a much simpler baseline you can just leave out a multiple layered graph neural network and just use the aggregation on the k-hop neighborhood so instead of using just a single a one hop neighborhood so your your direct connections you can use and do the the MPN and the message passing scheme I told you multiple times you can instead do the following you can just aggregate the neighbors from the k-hop neighborhood where K can be maybe three or four which means you can get up to like four under in their indirect connections and build up and then just use a single just so aggregate those and then use a single MPN layer to to build up a nice representations which can even achieve better results in those complex spatial methods like graph attention network etc and they are much more computationally efficient because they are so much simpler now now there is a nice parallel I can make and I will make here between the CNNs and this thing here so if you know anything about CNNs you know that back in 2012 when Alex net was introduced and produced this thing called image net moment Alex net had nine I think 11 by 11 kernels whereas so that's really broad compared to VGG which in 2015 I think basically I figured they figure out that using 3 by 3 kernels in much deeper architectures is actually beneficial so you can see the parallel between this and the CNNs and basically with the arrival of certain ideas such as residual connections such as better realization we were able to make deeper and deeper CNNs and keep those kernel kernels really small at 3 by 3 size and that turned out to be really good so much better than using large kernels and shallower networks on the other hand we can see here in graph ML they're using big kernels and shallow networks actually produces better results on certain classes of graphs not on everything sometimes you still need deep graph neural networks and just keep that in mind okay so those were the main methods and now the topics that are following are a bit more esoteric but none nothing less useful useful from the previous methods I just explained like embeddings and spectrum methods and spatial methods so DNN expressivity is really interesting so let me let me just explain what it is in a nutshell and it's actually fairly easy when you put it this way you see these two graphs here and you can see that they are different so how we also call that more more like a more formal term for that would be their non isomorphic and what what you ideally want to have is you want to have your graph neural network assign a different embedding to two different to two different graphs that makes sense to two non isomorphic graphs on the other hand if the graphs are isomorphic you want to assign them the same embedding and that's the ideal case so all of the methods I explained you so far like GCN graph convolutional network or get are actually confused by this simple example as you can see the the color histogram we have four reds here four reds here four blues four blues two greens and two greens and the way how for example graph classification works you just take all of these features you kind of do a mean and then you'll try to do some prediction and because the histograms are the same in this particular case that means we'll have the same embedding even though they are non isomorphic so that's the potential failure case and we want to avoid this and so by building more expressive models they'll know how to cope with cases like this one and that's the whole so the field of expressivity deals with this exact problem trying to assign the same label if the graphs are isomorphic otherwise assign different a different not label different embeddings okay so that's the whole concept and this graph isomorphism network paper first showed that by using that GCN and GATS and all of those have there is certain families of graphs which confuse which confuse those MPNN networks and later a lot of cool papers appeared that do some sub count that use sub structures to to to increase the expressivity or use multiple aggregation techniques like principal neighborhood aggregation paper etc but that's the basic idea I want you to take out of this so yeah just discriminating between the nine non isomorphic graphs papers on exciting gene applications feel free to go through these resources I already walked you through these so yeah just go at your own at your own pace and hopefully you'll find some interesting new papers additional papers here okay so handling dynamic graphs chapter okay all the methods I've explained so far are dealing with static graphs so what's static what's dynamic so dynamic graphs are those that evolve with time so either new edges or new nodes appear as the time progresses okay so one good example would be social network maybe you have Facebook or Twitter where each new tweet is a new node that appears at certain time point yeah the same goes with edges maybe somebody does a read feed and that's a new edge that appeared dynamically in this graph so the graph is constantly being pruned and increased so it's usually more it's usually always increasing but there is some pruning happening with somebody believes the account leads the tweet whatever for those types of graphs we need different methods and the two main ideas I want you to take out from from this section is that you need to find a way to represent the time which is a scalar as a vector and there are a couple of papers you can you can find so I've linked some papers like TGN here and you'll you'll see some references inside like time to back I think which shows one of the ways you can you can kind of so map the scalar the time into its vector eyes representation so that's one crucial idea that's needed and the second one is doing the concept of doing a temporal neighborhood so basically you have a single node let's say it's a tweet and what you want to do is you want to aggregate only those edges there may be at most one hour like old and you want to just ignore all of the other edges otherwise you have a huge graph and you have this thing called over squashing problem because you have so many nodes and you want to try to aggregate them usually doing some mean or something and that's kind of that's a similar effect happens in those bag of words if you're familiar with bag of words technique so you kind of know that by doing bag of roots and you lose the information same thing happens in RNNs where you get these this over squashing effect because as the sequence becomes longer you're kind of doing this bag of word thing and you're losing the older information so what so transformers for example tackled this problem by using the concept of attention right so you give more attention to more important feature vectors and ignore some less important ones so yeah basically temporal neighborhood helps us avoid that over squashing problems problem that happens in all of those bag of word methods such as in RNNs I linked a couple of unsupervised papers unsupervised learning papers like DJI and this variational graph autoencoder so if you're not familiar with variational autoencoder this paper probably won't make any sense it's just it gets three or four pages it's three or four pages long and it already assumes you know everything about variational autoencoders but it's a it's a nice read if you're interested in learning more about these unsupervised learning methods just make it I made a small tangent here on the this field called geometric deep learning so when you start doing graph ML you'll hear about geometric deep learning that's for sure and those are pretty much separate it's a pretty much separate research area which deals with manifolds and not with graphs per se so even though again you can treat certain manifolds once you discretize them them there they are basically graphs but here the same thing as with computer vision the same thing as with NLP you have a class of algorithms which are much better optimized to deal with manifolds even though they can be treated when discretized as graphs so just keep that in mind and don't make it too big of a detour into that field because it's pretty much separate research area compared to graph ML okay with that out of the way I linked a couple of cool visualization tools here like iGraph I've used iGraph personally in my graph attention network project check that one out if you haven't I'll link it down in the description as well and aside from that I've been mentioning data sets here and data sets are so important and they usually achieve not they usually don't get so much attention in the community usually you have something like ah there's this huge model like GPT that has like five through trillion parameters and that gets really hyped up but the data sets which are so important for progressing the research don't get so much attention and there is a problem with graph ML that people use these Cora sites here PPI and PubMed data sets a lot in the over the previous years and only recently have we started to integrate bigger real-world graphs and use them as benchmarks so what happened is that when you use these these let's call them toy benchmark graph data says like similar to amnesty computer vision you basically can't discern between much more expressive and better models from simpler baselines so there's the whole point of the benchmark right you want to you want to have a benchmark which clearly separates and fill sorts better algorithms from the worst ones and here the the the margin became smaller and smaller even between the simplest baselines and the most complex GNNs so we needed new data sets and so this I link this open graph benchmark which is a nice initiative that introduced much better much much larger and more diverse graphs into this graph ML field and that's a really good thing and we should be very grateful for people who invest their time to develop better benchmarks even though that's not as exciting as developing some proposing some novel novel idea novel it's usually research is usually super incremental rarely do you see something that's completely new like graph embedding methods Vertweck already existed they just adapted it to to graphs similarly for most of these methods are just a variation on some research that already existed just keep that in mind I linked some cool resources Michael Bronstein's got a really awesome blog highly recommend you go and read those but probably you'll want to leave those for a bit later because he's a professor and he has a that his blogs are a bit more technical so you want to start with those high level blogs read through resources I've listed and only then start reading Michael's blog blog finally once you get some once you get your foot inside graph ML start following people in Twitter and start following Sergey's newsletter I highly recommend it that's a good way to keep in sync with the newest things happening in a specific field so similarly to how Sergey has newsletter in graph ML Sebastian Huda has a really nice newsletter in the world of natural language processing so yeah this is something that was shown to be a nice format for transferring knowledge in the community so you should be following these small specialized newsletters highly recommended I'll skip through this I had I made a couple of videos on graph attention network project they did so we gained a lot of traction like it got 750 stars since I published it a couple of weeks ago and you can go through it you'll find it useful probably I've got both the trans active as well as the inductive examples inside like both the Cora which is trans active and PPI protein protein interaction data set which is inductive and I've got the I made this Jupiter notebook as well so yeah do check it out that's the project and that's pretty much everything I wanted to tell you maybe a short note on some other detours you could be making and that's I already mentioned geometric deep learning so it's a in a way it's a separate research topic so be aware don't don't just don't make a mistake and start reading those papers because because they are pretty much independent orthogonal to the graph ML papers then there is this equivalent deep learning subfield which deals with making your algorithms much more statistically efficient by exploiting symmetries of your problem so the same way as CNN's are translationally equivalent they are trying to make models which are also rotationally equivalent which can just exploit different symmetries and again it's a separate research subfield be aware of it don't don't waste too much time of going there if you just want to focus on graph ML even though again there is always some cross-semination of ideas so you have to draw the line where you want to explore more versus exploit more finally knowledge graphs there's something there are mix between NLP and graph ML so again totally new terminology totally new set of people history papers so just be cognizant of that fact that's it you know the drill hit that subscribe button hit the bell icon and until next time keep learning deep
[{"start": 0.0, "end": 4.16, "text": " What's up folks? So in this video I'm going to tell you more about how to get"}, {"start": 4.16, "end": 8.8, "text": " started with GraphML. So I'm wrapping up my exploration of Graph Machine"}, {"start": 8.8, "end": 12.44, "text": " Learning and I wrote this blog on how to get started. So I thought of just creating"}, {"start": 12.44, "end": 17.44, "text": " a video and maybe telling you more about my opinions and some of the stuff that"}, {"start": 17.44, "end": 21.88, "text": " I've already written in the blog and giving you some complimentary ideas. So"}, {"start": 21.88, "end": 26.2, "text": " yeah, basically stay tuned and hopefully you'll learn something new. Okay so the"}, {"start": 26.2, "end": 32.24, "text": " first thing I want to talk about is this. So research can be really tough and just"}, {"start": 32.24, "end": 37.08, "text": " be aware that everybody, every single like top-notch researcher you know"}, {"start": 37.08, "end": 41.56, "text": " of or engineer or successful business person, whatever, has hard days and it's"}, {"start": 41.56, "end": 45.16, "text": " not always easy. So I'm talking from my own experience. So it's not always"}, {"start": 45.16, "end": 51.14, "text": " mentally easy to stick with the plan and execute and do the research, read the"}, {"start": 51.14, "end": 56.120000000000005, "text": " papers, code up that project. So just be aware of it that you're not alone in"}, {"start": 56.12, "end": 59.919999999999995, "text": " this and you should never give up. That's my only advice I have to give you."}, {"start": 59.919999999999995, "end": 64.8, "text": " Never freaking give up. Okay? Secondly, don't get demotivated by the people who"}, {"start": 64.8, "end": 69.84, "text": " are, who seem, who appear to be super successful in something. So the thing is"}, {"start": 69.84, "end": 76.24, "text": " most of them are really specialized. So I like to think from about this from like"}, {"start": 76.24, "end": 80.03999999999999, "text": " this linear algebra perspective. So maybe nerdy but there is a method to this"}, {"start": 80.03999999999999, "end": 83.08, "text": " madness. Basically I want to say the following. So among all of those"}, {"start": 83.08, "end": 86.08, "text": " different dimensions, all of the things you could be doing researching, learning,"}, {"start": 86.08, "end": 91.52, "text": " about, let's take two axes. So one axis would be maybe somebody who is just"}, {"start": 91.52, "end": 95.44, "text": " reading research papers. So some fictive person that's only reading research. On"}, {"start": 95.44, "end": 100.44, "text": " the other hand you have people who are just coding something, who are working on"}, {"start": 100.44, "end": 104.24, "text": " some deep learning libraries, like maybe in the context of GraphML that would be"}, {"start": 104.24, "end": 109.16, "text": " PyTorch Geometric. And most people tend to be somewhere in between. But there are"}, {"start": 109.16, "end": 112.48, "text": " a lot of people who only read research paper and they never implement"}, {"start": 112.48, "end": 117.32000000000001, "text": " something from scratch or never work on anything that has to do with engineering"}, {"start": 117.32000000000001, "end": 122.12, "text": " or coding. And I want to tell you that they don't know many details even though"}, {"start": 122.12, "end": 126.52000000000001, "text": " they appear to understand everything because they curate their own content."}, {"start": 126.52000000000001, "end": 131.08, "text": " They don't know many details of how stuff works. And I'm talking from my"}, {"start": 131.08, "end": 134.84, "text": " personal experience. So until I start coding something up from scratch I"}, {"start": 134.84, "end": 139.66, "text": " didn't, I don't understand thoroughly how much I actually don't know, if that"}, {"start": 139.66, "end": 144.28, "text": " makes sense. So you want to be somewhere in between. If you want to be a really"}, {"start": 144.28, "end": 147.96, "text": " successful researcher, in my opinion, and you want to boost that H index if that's"}, {"start": 147.96, "end": 151.92, "text": " of something of interest to you, you should probably be reading, focusing on"}, {"start": 151.92, "end": 156.44, "text": " reading research papers a lot. But you should also make these, let's call them"}, {"start": 156.44, "end": 161.28, "text": " detours into implementing something and actually playing with the code. On the"}, {"start": 161.28, "end": 165.68, "text": " other hand if you want to be a great engineer and you probably want to focus"}, {"start": 165.68, "end": 170.28, "text": " on code but you still want to research and read, allocate some time to read"}, {"start": 170.28, "end": 175.20000000000002, "text": " research papers because otherwise you'll just lose the pace and you'll,"}, {"start": 175.20000000000002, "end": 181.36, "text": " the field is progressing very rapidly and you got to allocate some time to do"}, {"start": 181.36, "end": 187.84, "text": " the due diligence. Having said that, let me walk you through this blog. So first"}, {"start": 187.84, "end": 193.32, "text": " GraphML and hopefully you know what GraphML is. You heard about graphs so"}, {"start": 193.32, "end": 197.23999999999998, "text": " those are those data structures and GraphML is just a bunch of machine"}, {"start": 197.23999999999998, "end": 203.04, "text": " learning models so we learn on top of that graph data. And the first thing I"}, {"start": 203.04, "end": 206.35999999999999, "text": " want to advise you here is don't, if you're, especially if you're a"}, {"start": 206.35999999999999, "end": 211.28, "text": " perfectionist, you'll tend to be like this. I want to make a, I want to go on a"}, {"start": 211.28, "end": 215.64, "text": " tangent here and maybe invest three months of my time and get some solid"}, {"start": 215.64, "end": 222.12, "text": " background in graph theory, a branch of mathematics. Or maybe get a really good"}, {"start": 222.12, "end": 226.44, "text": " foundations in the network analysis field. And I want to tell you, you don't"}, {"start": 226.44, "end": 231.12, "text": " need to do that. You can jump straight into the field and make a first step and"}, {"start": 231.12, "end": 236.88, "text": " without doing all of those detours. So it's totally fine to start reading this"}, {"start": 236.88, "end": 243.16, "text": " blog and start with high-level resources and that would be your, as a first step"}, {"start": 243.16, "end": 249.4, "text": " to on your journey through Graph Machine Learning. Sometimes you may want to take"}, {"start": 249.4, "end": 255.32, "text": " a step back or maybe better said, make a side step and invest maybe three"}, {"start": 255.32, "end": 259.64, "text": " months of your time into building up more solid mathematical"}, {"start": 259.64, "end": 266.28000000000003, "text": " foundations. And me personally, last year in 2020, I did maybe, I invested maybe"}, {"start": 266.28000000000003, "end": 270.2, "text": " around three, four months into reading this book, Mathematics for Machine"}, {"start": 270.2, "end": 274.48, "text": " Learning. And it's a bit complex if you're intermediate or even"}, {"start": 274.48, "end": 280.64000000000004, "text": " beginner. But it meant a lot for me and it helped me better understand the"}, {"start": 280.64000000000004, "end": 286.12, "text": " stuff that I later learned. So the strategy of doing side steps and"}, {"start": 286.12, "end": 293.0, "text": " investing into your some foundations is also a good idea sometimes. So that's"}, {"start": 293.0, "end": 297.76, "text": " GraphML. You have machine learning, you have machine learning models that learn"}, {"start": 297.76, "end": 303.24, "text": " on graph data. Instead of images or text like in computer vision or NLP, you're"}, {"start": 303.24, "end": 309.36, "text": " learning from graphs. And now before we start, before I show you the resources,"}, {"start": 309.36, "end": 313.36, "text": " let's just, let me just walk you through some cool applications. And one more"}, {"start": 313.36, "end": 318.88, "text": " thing to note, you'll hear a lot about graph neural networks here and for the"}, {"start": 318.88, "end": 323.40000000000003, "text": " sake of, for now just think of them as neural networks that work with graphs"}, {"start": 323.40000000000003, "end": 328.76, "text": " and somehow can process those graphs and do the classification or"}, {"start": 328.76, "end": 333.92, "text": " regression on nodes or edges or even the complete graphs. So just have them as a"}, {"start": 333.92, "end": 340.71999999999997, "text": " black box on top of your mind, okay? So first things first, GraphML is"}, {"start": 340.71999999999997, "end": 346.36, "text": " such an exciting topic because the applications you'll see here are so much"}, {"start": 346.36, "end": 351.0, "text": " different from those you'll see in the computer vision or the NLP worlds. So"}, {"start": 351.0, "end": 355.03999999999996, "text": " there in computer vision you'll see, you'll be seeing algorithms that"}, {"start": 355.04, "end": 361.24, "text": " manipulate images or videos and in NLP you'll be seeing algorithms that"}, {"start": 361.24, "end": 366.40000000000003, "text": " manipulate text and those are all pretty homogeneous. So even though you can do, I"}, {"start": 366.40000000000003, "end": 372.12, "text": " don't know, facial landmark detection, pose estimation, bunch of things,"}, {"start": 372.12, "end": 377.84000000000003, "text": " segmentation, regressing 3D models, bunch of things in computer vision, they are"}, {"start": 377.84000000000003, "end": 383.52000000000004, "text": " all still kind of related. And then when you start doing GraphML you understand"}, {"start": 383.52, "end": 389.59999999999997, "text": " like all the plethora, like the vast diversity of different applications"}, {"start": 389.59999999999997, "end": 395.12, "text": " you can do with graphs. So two cool applications I start, I want to start"}, {"start": 395.12, "end": 401.24, "text": " with our recommendation systems and if you're not aware already, like in my"}, {"start": 401.24, "end": 406.35999999999996, "text": " opinion, recommendation systems are the single most important and influential"}, {"start": 406.35999999999996, "end": 412.08, "text": " piece of AI currently in 2021. So you interact with with recommendation systems"}, {"start": 412.08, "end": 418.44, "text": " all the time, YouTube recommendation, Facebook, Twitter, Pinterest, whatever,"}, {"start": 418.44, "end": 424.24, "text": " just name it, and you have recommendation systems influencing your decisions. So"}, {"start": 424.24, "end": 428.84, "text": " before I started making YouTube videos I wasn't fully aware of it and now I'm"}, {"start": 428.84, "end": 432.96, "text": " getting a thorough understanding of how how influential recommender systems are."}, {"start": 432.96, "end": 439.08, "text": " So basically as an example I've seen many videos which are really top-notch,"}, {"start": 439.08, "end": 445.32, "text": " high quality, that have 5k views and not because the presenter was anything less"}, {"start": 445.32, "end": 451.4, "text": " eloquent or the content was worse, no. It was just because he or she came on later"}, {"start": 451.4, "end": 456.96, "text": " and so he or she is not on the on the front page, like among the first"}, {"start": 456.96, "end": 462.96, "text": " queries. And then you'll see some videos which were made maybe 2015, 16, 17 and"}, {"start": 462.96, "end": 467.15999999999997, "text": " they have amassed a lot of views and so that's what YouTube will recommend you."}, {"start": 467.16, "end": 472.76000000000005, "text": " So what I've learned is I now tend to go past the first five results before I"}, {"start": 472.76000000000005, "end": 477.28000000000003, "text": " would just open the most viewed videos but now I'm completely aware how"}, {"start": 477.28000000000003, "end": 482.8, "text": " recommendation systems can actually bring you inferior recommendations,"}, {"start": 482.8, "end": 487.56, "text": " unfortunately. And yeah, engineers and researchers are working on this and one"}, {"start": 487.56, "end": 492.52000000000004, "text": " way is to just kind of use graph neural networks. So because the problem as you"}, {"start": 492.52, "end": 497.59999999999997, "text": " can see here with with uBreads for example, you can really nicely model all"}, {"start": 497.59999999999997, "end": 503.28, "text": " of these problems as graphs. So here you have a graph between users and food and"}, {"start": 503.28, "end": 509.32, "text": " uBreads has a separate graph between users and restaurants. They still haven't"}, {"start": 509.32, "end": 512.4399999999999, "text": " merged the two together even though I think they're currently working on it or"}, {"start": 512.4399999999999, "end": 516.4399999999999, "text": " maybe they even maybe they recently even deployed it. I'm not aware of it but yeah"}, {"start": 516.4399999999999, "end": 521.88, "text": " this is the last news I'm aware of. And so just processing this graph data"}, {"start": 521.88, "end": 526.56, "text": " you can make better recommendations to your users and"}, {"start": 526.56, "end": 531.72, "text": " obviously I won't get into all of the details. I did some videos on Pinsage,"}, {"start": 531.72, "end": 536.84, "text": " that's the recommendation engine at Pinterest and you can check out that one"}, {"start": 536.84, "end": 544.0, "text": " somewhere here in the card. Okay, so you can kind of see how the graph neural"}, {"start": 544.0, "end": 549.04, "text": " network Pinsage improve upon these baselines. Here you can see people"}, {"start": 549.04, "end": 552.76, "text": " carrying some logs and here you can see the visual. If you just take the visual"}, {"start": 552.76, "end": 557.8399999999999, "text": " baseline it will bring you images that are like really similar when it comes to"}, {"start": 557.8399999999999, "end": 562.0, "text": " appearance because they are also grayscale but you can see the semantics"}, {"start": 562.0, "end": 565.8, "text": " like here the soldiers and here you can see actually the trees and the logging"}, {"start": 565.8, "end": 572.28, "text": " so Pinsage got both the semantics as well as the appearance successfully by"}, {"start": 572.28, "end": 576.4399999999999, "text": " leveraging both the graph structure as well as the visual textual information"}, {"start": 576.44, "end": 582.44, "text": " that's available in those graphs on those platforms. So those are some nice"}, {"start": 582.44, "end": 588.5200000000001, "text": " applications and aside from from recommendation systems we have these"}, {"start": 588.5200000000001, "end": 595.8000000000001, "text": " basically fake news detection GNNs and that's so important. The societal"}, {"start": 595.8000000000001, "end": 601.32, "text": " benefits of deploying really successful fake news detection algorithms are so"}, {"start": 601.32, "end": 605.24, "text": " important and again you can see here maybe on Twitter you have a"}, {"start": 605.24, "end": 610.12, "text": " single tweet and somebody retweets a tweet that would be a second node and you"}, {"start": 610.12, "end": 614.96, "text": " have the edge and so you can build up the graph and basically these networks"}, {"start": 614.96, "end": 620.2, "text": " can figure out the propagation patterns of those retweets and just by looking at"}, {"start": 620.2, "end": 625.5600000000001, "text": " the propagation patterns it was shown in previous research that we can figure out"}, {"start": 625.5600000000001, "end": 630.72, "text": " whether the news is actually fake or real and again that's a really naturally"}, {"start": 630.72, "end": 636.8000000000001, "text": " model as a graph so GNNs to the rescue. Aside from this app there is one"}, {"start": 636.8000000000001, "end": 641.76, "text": " more cool app which is basically figuring out the probability that you"}, {"start": 641.76, "end": 648.0, "text": " will produce certain event depending on your neighborhood and whether they did"}, {"start": 648.0, "end": 654.72, "text": " the same thing so for example to be less abstract maybe the goal is to figure"}, {"start": 654.72, "end": 659.24, "text": " out whether you'll retweet a tweet if some of your neighbors retweeted the"}, {"start": 659.24, "end": 663.48, "text": " same tweet that would be one concrete example and again you have the graph"}, {"start": 663.48, "end": 669.44, "text": " structure and you can leverage to regress these probabilities so various"}, {"start": 669.44, "end": 674.72, "text": " cool applications and what's even more interesting is that graph ML so all of"}, {"start": 674.72, "end": 679.8, "text": " these are still kind of familiar to you but like graph ML goes into these deep"}, {"start": 679.8, "end": 686.0, "text": " fundamental science problems and can naturally solve them so one of them"}, {"start": 686.0, "end": 691.92, "text": " would be this example I gave with the neutrino detection where basically the"}, {"start": 691.92, "end": 696.4, "text": " setup is the following you have a huge huge cube of ice and you have these"}, {"start": 696.4, "end": 700.96, "text": " light sensors inter dispersed in the like distributed throughout this ice and"}, {"start": 700.96, "end": 706.24, "text": " what happens is a neutrino comes comes out from the space and hits the ice and"}, {"start": 706.24, "end": 710.88, "text": " interacts with the ice molecules and in that interaction something that's called"}, {"start": 710.88, "end": 716.08, "text": " charank of light emits and basically those sensors those light sensors can catch"}, {"start": 716.08, "end": 721.6, "text": " those that light and you can basically see certain nodes of that grid like"}, {"start": 721.6, "end": 726.92, "text": " structure of sensors get activated and then you can just deploy the graph neural"}, {"start": 726.92, "end": 732.36, "text": " network on top to figure out whether the neutrino came from some certain events"}, {"start": 732.36, "end": 737.04, "text": " we care about like supernova explosions or whatever so I don't know about you"}, {"start": 737.04, "end": 740.9599999999999, "text": " but like this completely blew out my mind because usually you see neural"}, {"start": 740.9599999999999, "end": 745.04, "text": " networks applied to images videos text and that's it and here you can see"}, {"start": 745.04, "end": 751.68, "text": " fundamental science and GNN is being used to solve them to solve those"}, {"start": 751.68, "end": 756.52, "text": " problems okay yeah there are many other apps I'll skip some of them like"}, {"start": 756.52, "end": 761.1999999999999, "text": " figuring out the molecules that can help beat cancer because you can see you can"}, {"start": 761.2, "end": 767.6400000000001, "text": " model the the thing naturally as a graph again you have genes you have proteins"}, {"start": 767.6400000000001, "end": 772.72, "text": " all of those interactions and here's some molecules or drugs and you you model"}, {"start": 772.72, "end": 777.1600000000001, "text": " that as a graph and you can find interesting information like which"}, {"start": 777.1600000000001, "end": 781.84, "text": " molecules can help treat cancer better or prevent cancer then you have drug"}, {"start": 781.84, "end": 786.6400000000001, "text": " side effects when you combine multiple drugs you want to be sure it would be"}, {"start": 786.64, "end": 791.64, "text": " beneficial if we could computationally calculate at the possible side effects"}, {"start": 791.64, "end": 796.0, "text": " because the combinatorics is so huge here and it's impractical to do it in a"}, {"start": 796.0, "end": 804.48, "text": " lab setting in vivo we need to do it in in silica or computationally so many"}, {"start": 804.48, "end": 809.88, "text": " cool applications so the last one I'll note here is the one that it was gained"}, {"start": 809.88, "end": 816.04, "text": " a lot of traction last year in 2020 basically we as a society have a huge"}, {"start": 816.04, "end": 822.64, "text": " problem with with anti antibiotic resistant bacteria so the reason that"}, {"start": 822.64, "end": 826.5999999999999, "text": " happens is because people are misusing antibiotics they're using that them for"}, {"start": 826.5999999999999, "end": 831.56, "text": " viruses they're using them to treat common colds and you shouldn't be using"}, {"start": 831.56, "end": 835.5999999999999, "text": " antibiotics in those cases you should be using antibiotic a specific antibiotic"}, {"start": 835.5999999999999, "end": 839.54, "text": " if you're sure you have a certain bacteria that this antibiotic can"}, {"start": 839.54, "end": 846.52, "text": " inhibit and because of all of those misuse there is a threat of this"}, {"start": 846.52, "end": 852.56, "text": " bacteria which is resistant to every single antibiotic and that's like the"}, {"start": 852.56, "end": 857.52, "text": " worst nightmare in the like in the science community and in the medicine in"}, {"start": 857.52, "end": 863.24, "text": " general pharmacy and so what this graph neural network did is it was trained on"}, {"start": 863.24, "end": 869.72, "text": " a set of these of these specific molecules and then and was trained to"}, {"start": 869.72, "end": 875.16, "text": " figure out which of these molecules tend to inhibit one specific bacteria namely"}, {"start": 875.16, "end": 880.2, "text": " the asharica coli in this example and then once they deployed that trained"}, {"start": 880.2, "end": 885.8, "text": " graph neural network onto a specific new subset a new data set it was able to"}, {"start": 885.8, "end": 891.64, "text": " filter out hundred candidates that gave a high response a high probability that"}, {"start": 891.64, "end": 898.16, "text": " this molecule will inhibit asharica coli and then they just dedicated that"}, {"start": 898.16, "end": 903.56, "text": " small small subset of molecules to scientists and they were able to find"}, {"start": 903.56, "end": 908.92, "text": " this molecule called helicin which basically can treat which was basically"}, {"start": 908.92, "end": 914.48, "text": " found out to be a super potent antibiotic and extremely successful in"}, {"start": 914.48, "end": 919.92, "text": " inhibiting asharica coli and previously it was just they considered this same"}, {"start": 919.92, "end": 928.16, "text": " molecule for his for its anti-diabetic properties so those are just some of the"}, {"start": 928.16, "end": 933.68, "text": " super exciting applications of graph neural networks and I hope you you found"}, {"start": 933.68, "end": 940.3199999999999, "text": " this interesting and and that it motivated you to further investigate the"}, {"start": 940.3199999999999, "end": 943.12, "text": " graph ML field"}, {"start": 943.12, "end": 951.04, "text": " now let me just kind of make a brief detour and compare graph ML with"}, {"start": 951.04, "end": 954.72, "text": " computer vision and natural language processing and I personally think it's"}, {"start": 954.72, "end": 960.0, "text": " really important to to try and connect the ideas from various subfields of deep"}, {"start": 960.0, "end": 964.96, "text": " learning because by doing that but we're just creating those connections I think"}, {"start": 964.96, "end": 971.52, "text": " you'll be better to generate much better research ideas and yeah it's just cool"}, {"start": 971.52, "end": 975.6, "text": " to kind of make this connection just gives you some kind of satisfaction as"}, {"start": 975.6, "end": 982.24, "text": " well so basically you can treat computer vision or images and videos in general"}, {"start": 982.24, "end": 987.24, "text": " as a just a special case of graph which is highly regular and we call these"}, {"start": 987.24, "end": 994.0799999999999, "text": " grids so you can see here a four by four pixel image can be represented as this"}, {"start": 994.0799999999999, "end": 999.16, "text": " specific graph and here we have four neighbors in general you can have up to"}, {"start": 999.16, "end": 1002.88, "text": " eight neighbors because right you have the diagonal pixels as well and you have"}, {"start": 1002.88, "end": 1008.0799999999999, "text": " the top left top right bottom left bottom right and that means we can"}, {"start": 1008.0799999999999, "end": 1012.52, "text": " actually deploy graph neural networks to solve computer vision problems and same"}, {"start": 1012.52, "end": 1017.12, "text": " goes for for NLP because sentence is just basically you can treat sentence"}, {"start": 1017.12, "end": 1023.8, "text": " either as a fully connected graph or maybe just as a as a tree and yeah but"}, {"start": 1023.8, "end": 1027.24, "text": " the thing I want to mention here and I mentioned that in the blog as well is"}, {"start": 1027.24, "end": 1031.64, "text": " don't fall into the trap into thinking that you can you can better solve these"}, {"start": 1031.64, "end": 1036.72, "text": " problems with graph ML so the thing is certain biases that CNN's have like"}, {"start": 1036.72, "end": 1046.4, "text": " having weight sharing and localization are super useful and we show that that's"}, {"start": 1046.4, "end": 1051.36, "text": " much more efficient than using graph ML in some cases though you can use graph"}, {"start": 1051.36, "end": 1056.0, "text": " neural networks and perform better so when I say that I mean the following"}, {"start": 1056.0, "end": 1060.68, "text": " transformers are a special case of graph neural networks in a sense so once you"}, {"start": 1060.68, "end": 1065.48, "text": " have a fully connected graph transformer is basically a graph neural network"}, {"start": 1065.48, "end": 1071.68, "text": " that operates on a fully connected graph and it was shown in 2020 in October I"}, {"start": 1071.68, "end": 1077.96, "text": " think the vision transformer that we can use transformers on images to improve"}, {"start": 1077.96, "end": 1084.4, "text": " upon CNN's if we have enough data so if you have a huge amount of data it is"}, {"start": 1084.4, "end": 1089.1200000000001, "text": " possible maybe to use transformers or graph neural networks to achieve even"}, {"start": 1089.1200000000001, "end": 1093.4, "text": " better results then by using CNN's but there's a saying if you only have hammer"}, {"start": 1093.4, "end": 1098.0, "text": " everything looks like a nail and it's it's really bad if you if you just have"}, {"start": 1098.0, "end": 1102.6000000000001, "text": " the graph ML context and you don't have if you don't know other fields you're"}, {"start": 1102.6000000000001, "end": 1107.0400000000002, "text": " gonna try and and and use graph neural network for everything same goes for"}, {"start": 1107.0400000000002, "end": 1111.1200000000001, "text": " deep learning and that's why it's useful to be aware of different fields"}, {"start": 1111.12, "end": 1115.1399999999999, "text": " different methods different techniques because for every specific problem there"}, {"start": 1115.1399999999999, "end": 1120.4399999999998, "text": " is a solution that's highly optimized to be used for that specific problem now"}, {"start": 1120.4399999999998, "end": 1126.0, "text": " there is a trade-off between being as general as possible like maybe GPT and"}, {"start": 1126.0, "end": 1129.6, "text": " on the other hand having a super specialized algorithms so they are better"}, {"start": 1129.6, "end": 1133.32, "text": " they are more specialized they are more optimized but then you have the overhead"}, {"start": 1133.32, "end": 1139.4399999999998, "text": " of some like developers having to invest a lot of their time to now specialize"}, {"start": 1139.44, "end": 1144.3200000000002, "text": " that's to tune that algorithm for every single problem encounter on the other"}, {"start": 1144.3200000000002, "end": 1148.6000000000001, "text": " hand you have this general architectures you just apply them and they work on"}, {"start": 1148.6000000000001, "end": 1152.76, "text": " everything you don't have to invest any more time but they are suboptimal in"}, {"start": 1152.76, "end": 1157.3200000000002, "text": " every single of those problems so that's the trade-off having said all of that"}, {"start": 1157.3200000000002, "end": 1163.0800000000002, "text": " let me now walk you through the resources in the way and just kind of"}, {"start": 1163.0800000000002, "end": 1167.24, "text": " advise you how you can structure your learning of graph machine learning okay"}, {"start": 1167.24, "end": 1172.6, "text": " so if I were you I'd first start with graph embedding methods so these are not"}, {"start": 1172.6, "end": 1177.08, "text": " graph neural networks these are a specific class of methods which are"}, {"start": 1177.08, "end": 1182.42, "text": " similar to Vertubec if you're familiar with natural language processing and you"}, {"start": 1182.42, "end": 1187.28, "text": " know what Vertubec is these are going to be much more familiar and easy to"}, {"start": 1187.28, "end": 1192.48, "text": " grasp so they are just repurposed Vertubec onto graphs so basically for"}, {"start": 1192.48, "end": 1197.32, "text": " example you have these these algorithms DeepWalk and No2vec and Planetoid and what"}, {"start": 1197.32, "end": 1203.76, "text": " Deep2Walk does is the following it just samples random walks from a graph and"}, {"start": 1203.76, "end": 1208.72, "text": " you can treat those as a sentence so if you treat the graph as a corpus of text"}, {"start": 1208.72, "end": 1214.44, "text": " and you treat that specific random walk as as a sample sentence it's basically"}, {"start": 1214.44, "end": 1219.8, "text": " the same thing as Vertubec you just want to make sure that the neighboring words"}, {"start": 1219.8, "end": 1224.76, "text": " have similar embeddings so if they tend to be close on those in those sentences"}, {"start": 1224.76, "end": 1228.96, "text": " you want to make the embeddings really close and if they're if they never"}, {"start": 1228.96, "end": 1232.8, "text": " appear in the same sentence if some words never appear in the same sentence"}, {"start": 1232.8, "end": 1238.2, "text": " you want to make their embeddings really different so that's the main idea of all"}, {"start": 1238.2, "end": 1244.82, "text": " of these methods so take some time to kind of understand these because all of"}, {"start": 1244.82, "end": 1249.6799999999998, "text": " these graph embedding methods can be used for exciting graph problems like"}, {"start": 1249.6799999999998, "end": 1254.4399999999998, "text": " doing regression on top of those embeddings or classification both on"}, {"start": 1254.4399999999998, "end": 1261.0, "text": " node wise level edgewise and graph wise level so I should you should probably"}, {"start": 1261.0, "end": 1264.9199999999998, "text": " start with these and that's gonna be it's gonna be a nice first step okay in"}, {"start": 1264.9199999999998, "end": 1269.28, "text": " a way it's orthogonal to the graph neural networks that I'm going to tell"}, {"start": 1269.28, "end": 1275.6399999999999, "text": " you about more in a second so graph neural networks when it comes to graph"}, {"start": 1275.6399999999999, "end": 1280.6399999999999, "text": " neural networks there are two main branches and one is the spectral methods"}, {"start": 1280.6399999999999, "end": 1286.48, "text": " and the second one is spatial or message passing methods and my advice here is to"}, {"start": 1286.48, "end": 1292.8799999999999, "text": " take some time and investigate spectral methods because the terminology and"}, {"start": 1292.8799999999999, "end": 1298.48, "text": " references will get get propagated into the spatial methods and you'll be seeing"}, {"start": 1298.48, "end": 1301.6, "text": " some terminology from the spectral methods so it's good to kind of"}, {"start": 1301.6, "end": 1307.0, "text": " understand at least some basics about these so having said that if you're not"}, {"start": 1307.0, "end": 1312.28, "text": " familiar if you don't have a strong base in like fundamentals in linear algebra"}, {"start": 1312.28, "end": 1315.84, "text": " if you don't understand what eigen space is if you don't understand what eigen"}, {"start": 1315.84, "end": 1320.04, "text": " vector and eigenvalue is if you don't have some understanding of the Fourier"}, {"start": 1320.04, "end": 1325.9, "text": " analysis or signal processing this is going to be tough like you should"}, {"start": 1325.9, "end": 1330.0, "text": " probably just read some high-level blogs and try to kind of get familiar with"}, {"start": 1330.0, "end": 1335.4, "text": " the words and terminology and don't kind of pull yourself down just because you"}, {"start": 1335.4, "end": 1340.5400000000002, "text": " don't understand everything because yeah as I told you you'll need some time to"}, {"start": 1340.5400000000002, "end": 1345.64, "text": " understand it especially if you don't have the fundamentals but having said"}, {"start": 1345.64, "end": 1351.8000000000002, "text": " that just go through it get some feeling because that's gonna be super useful for"}, {"start": 1351.8, "end": 1356.68, "text": " you believe me so the reason I tell you not to worry about them too much is"}, {"start": 1356.68, "end": 1361.6399999999999, "text": " because they are computationally very expensive because how they work you have"}, {"start": 1361.6399999999999, "end": 1365.32, "text": " something called graph ML so they're just a huge matrix and what you want to"}, {"start": 1365.32, "end": 1370.56, "text": " do is you want to calculate those eigenvectors and that operation is very"}, {"start": 1370.56, "end": 1379.6399999999999, "text": " expensive so and not only that it's that also these these methods are key and"}, {"start": 1379.64, "end": 1383.0800000000002, "text": " generalized to different graph structures so they they have to be"}, {"start": 1383.0800000000002, "end": 1386.92, "text": " applied in a transductive manner and can be applied to new graphs which are"}, {"start": 1386.92, "end": 1392.18, "text": " smaller bigger because they inherently are transductive so that's the second"}, {"start": 1392.18, "end": 1398.5600000000002, "text": " pin point of these algorithms but still I told you take some time and learn about"}, {"start": 1398.5600000000002, "end": 1408.48, "text": " them okay in my opinion again in this video is full of opinions you want to"}, {"start": 1408.48, "end": 1412.48, "text": " start with as always with a top-to-bottom approach you want to start"}, {"start": 1412.48, "end": 1416.72, "text": " with some high-level videos high-level blocks that will get you exposed to the"}, {"start": 1416.72, "end": 1421.96, "text": " terminology and new ideas new concepts without you having to understand every"}, {"start": 1421.96, "end": 1425.72, "text": " single detail so you're just trying to build up that skeleton of knowledge and"}, {"start": 1425.72, "end": 1430.68, "text": " you should probably start doing that by by watching videos and reading through"}, {"start": 1430.68, "end": 1435.16, "text": " some medium and towards data science blocks so those are usually really"}, {"start": 1435.16, "end": 1441.28, "text": " nicely curated high quality blogs and videos and they'll get you exposed to"}, {"start": 1441.28, "end": 1448.16, "text": " the essential concepts having said that you'll find some of the links here and"}, {"start": 1448.16, "end": 1453.28, "text": " this video from the from the New York University they are they're kind of"}, {"start": 1453.28, "end": 1461.0800000000002, "text": " publishing all of the lectures they hold there so that's super super super"}, {"start": 1461.08, "end": 1466.48, "text": " useful for the community and yeah you should start with those I mentioned the"}, {"start": 1466.48, "end": 1471.1599999999999, "text": " Fourier analysis and stuff so if you don't know about if you haven't watched"}, {"start": 1471.1599999999999, "end": 1475.3, "text": " this one even if you know everything about Fourier I strongly suggest you go"}, {"start": 1475.3, "end": 1479.84, "text": " ahead and watch Grant Sanderson's video on Fourier so when I say Grant"}, {"start": 1479.84, "end": 1483.8, "text": " Sanderson's he's better known as the three blue one brown channel and he's"}, {"start": 1483.8, "end": 1488.84, "text": " got some a couple of videos on Fourier which are so nicely visualized and if"}, {"start": 1488.84, "end": 1493.28, "text": " you don't have the background from Fourier analysis this is probably going"}, {"start": 1493.28, "end": 1496.9199999999998, "text": " to be enough like a high-level understanding of what Fourier is"}, {"start": 1496.9199999999998, "end": 1502.6399999999999, "text": " basically decomposing your signal into a sum of differently scaled frequencies"}, {"start": 1502.6399999999999, "end": 1508.1999999999998, "text": " and it's gonna be a really good step for you to understand spectrum methods if"}, {"start": 1508.1999999999998, "end": 1514.08, "text": " you don't have any background whatsoever so do watch that one I've linked up to"}, {"start": 1514.08, "end": 1518.12, "text": " important papers here this one is the most important one spectral networks and"}, {"start": 1518.12, "end": 1523.4399999999998, "text": " locally connected networks and graphs and Chabnets just kind of localized that"}, {"start": 1523.4399999999998, "end": 1528.32, "text": " method local when I say localized I mean you just want to update a single"}, {"start": 1528.32, "end": 1531.84, "text": " node you'll just use the k-hop not neighborhood instead of using every"}, {"start": 1531.84, "end": 1535.8, "text": " single node in the graph which can be huge like in some examples when you're"}, {"start": 1535.8, "end": 1540.08, "text": " having maybe social level graphs you want to make those algorithms localized"}, {"start": 1540.08, "end": 1546.32, "text": " and the second thing aside from localization Chabnets made it also more"}, {"start": 1546.32, "end": 1550.8, "text": " efficient to calculate these representations by just leveraging the"}, {"start": 1550.8, "end": 1556.96, "text": " laplacian graph matrices and not and not calculating the eigenvectors I"}, {"start": 1556.96, "end": 1563.76, "text": " previously mentioned again some higher-level blogs just go through those"}, {"start": 1563.76, "end": 1569.36, "text": " research resources you should probably literally sequentially be learning about"}, {"start": 1569.36, "end": 1575.0, "text": " these resources I've linked you should probably read the graph embedding"}, {"start": 1575.0, "end": 1579.76, "text": " methods after you do the high-level blogs and videos but whatever feel free"}, {"start": 1579.76, "end": 1583.12, "text": " to experiment there but you should roughly be following the flow of the"}, {"start": 1583.12, "end": 1589.34, "text": " blog and as the blog progresses we go into more research and more niche topics"}, {"start": 1589.34, "end": 1597.88, "text": " like dynamics dynamic graphs and like graph expressivity etc okay having said"}, {"start": 1597.88, "end": 1602.92, "text": " that let me jump to spatial methods and these are the methods you'll be seeing"}, {"start": 1602.92, "end": 1608.4, "text": " you'll mostly be seeing so those are networks such as GCN graph convolutional"}, {"start": 1608.4, "end": 1612.8400000000001, "text": " network networks such as get graph attention network then you have graph"}, {"start": 1612.8400000000001, "end": 1616.8400000000001, "text": " sage pin sage many different spatial methods and on the high level they all"}, {"start": 1616.8400000000001, "end": 1621.44, "text": " work really similarly so basically you do the following thing so you have a"}, {"start": 1621.44, "end": 1625.8000000000002, "text": " node and you have the neighborhood and what you do is you transform the"}, {"start": 1625.8000000000002, "end": 1629.3200000000002, "text": " neighborhood feature vector somehow maybe doing some linear projection or"}, {"start": 1629.32, "end": 1634.8, "text": " MLP whatever you kind of is you somehow aggregate them maybe doing mean of"}, {"start": 1634.8, "end": 1639.6, "text": " those feature vectors or some more max or whatever and then you combine the"}, {"start": 1639.6, "end": 1644.4399999999998, "text": " central node its representation with that aggregated transform representation"}, {"start": 1644.4399999999998, "end": 1648.56, "text": " again somehow and that's your new representation and that's the general"}, {"start": 1648.56, "end": 1653.9199999999998, "text": " framework of message passing networks and all of these can be considered as"}, {"start": 1653.9199999999998, "end": 1658.08, "text": " special case of that framework I just explained you I covered many of these"}, {"start": 1658.08, "end": 1662.6399999999999, "text": " spatial methods on my youtube channel so check them out graph attention networks"}, {"start": 1662.6399999999999, "end": 1667.6, "text": " graph convolutional networks pin sage and graph sage so those are the two that"}, {"start": 1667.6, "end": 1671.84, "text": " were deployed in the recommendation system I previously mentioned at the"}, {"start": 1671.84, "end": 1675.28, "text": " beginning of the video message passing a neural network is the one that"}, {"start": 1675.28, "end": 1679.3999999999999, "text": " introduced this general framework general notion and understanding that"}, {"start": 1679.3999999999999, "end": 1683.8, "text": " all of these methods are just special case of something and they're doing"}, {"start": 1683.8, "end": 1688.36, "text": " something fairly similar to each other two more papers you should probably"}, {"start": 1688.36, "end": 1694.8799999999999, "text": " check out is the gated graph neural network which kinda popularized the"}, {"start": 1694.8799999999999, "end": 1699.6, "text": " usage of graph neural networks again because they were initially the similar"}, {"start": 1699.6, "end": 1706.8, "text": " methods were proposed already I think in 2004 2009 period time frame and this"}, {"start": 1706.8, "end": 1712.08, "text": " paper kind of popularized GNN's again and it's it's a nice paper it's it's"}, {"start": 1712.08, "end": 1719.12, "text": " really really general because it also includes the edge features it uses the"}, {"start": 1719.12, "end": 1725.12, "text": " group the LSTM for the aggregation I just mentioned and so it's a it's a it's"}, {"start": 1725.12, "end": 1728.08, "text": " a nice read aside from that paper learning convolutional neural networks"}, {"start": 1728.08, "end": 1733.6399999999999, "text": " for graphs is a cool paper because I didn't mention this but like the"}, {"start": 1733.6399999999999, "end": 1737.84, "text": " spectral and spatial methods what they try to do is generalize the concept of"}, {"start": 1737.84, "end": 1741.9199999999998, "text": " convolution that works so nicely in the computer vision world so with"}, {"start": 1741.92, "end": 1748.28, "text": " CNN's on two graphs and that turned out to be quite a difficult challenging"}, {"start": 1748.28, "end": 1753.0800000000002, "text": " problem in a way and so there were different approaches one was a spectral"}, {"start": 1753.0800000000002, "end": 1756.5600000000002, "text": " one one was a spatial one which is basically approximation of the spectral"}, {"start": 1756.5600000000002, "end": 1760.8400000000001, "text": " methods so it's not as mathematically rigorous as the spectral methods but a"}, {"start": 1760.8400000000001, "end": 1766.24, "text": " as I already mentioned in practice they work really nice and this paper here"}, {"start": 1766.24, "end": 1773.84, "text": " just introduced another idea and that was to basically copy the concept from"}, {"start": 1773.84, "end": 1778.64, "text": " the CNN's more literally than mathematically if that makes sense so"}, {"start": 1778.64, "end": 1783.16, "text": " basically what they try to do is introduce the concept of node ordering"}, {"start": 1783.16, "end": 1788.72, "text": " into into their method because the main problem with graphs and the reasons you"}, {"start": 1788.72, "end": 1793.76, "text": " can't you just can't use a fixed kernel like using the CNN is first of all you"}, {"start": 1793.76, "end": 1797.48, "text": " have the varying number of nodes sometimes a node will have only a single"}, {"start": 1797.48, "end": 1801.92, "text": " neighbor sometimes it will have thousand neighbors and you obviously can't use a"}, {"start": 1801.92, "end": 1807.08, "text": " nine by nine kernel or three by three kernel as it's commonly used in CNN's so"}, {"start": 1807.08, "end": 1811.52, "text": " that's the first problem the second problem you don't have any concept any"}, {"start": 1811.52, "end": 1816.72, "text": " any notion of ordering so like in CNN's you have top left pixel you have"}, {"start": 1816.72, "end": 1821.8, "text": " top right pixel in GNN's you don't have that notion so what they try to do is"}, {"start": 1821.8, "end": 1825.1599999999999, "text": " they try to linearize a neighborhood by doing something called graph labeling"}, {"start": 1825.1599999999999, "end": 1832.6399999999999, "text": " and then apply this the kernel that's always fixed onto those order linearized"}, {"start": 1832.6399999999999, "end": 1838.32, "text": " neighborhoods and so it's a nice read the idea itself didn't get that much"}, {"start": 1838.32, "end": 1844.72, "text": " traction but it's it's an awesome read and so do check it out okay I'm wrapping"}, {"start": 1844.72, "end": 1848.6, "text": " now with message passing networks with this one simplifying graph convolutional"}, {"start": 1848.6, "end": 1854.56, "text": " networks paper what they figured out is there are some graph classes like"}, {"start": 1854.56, "end": 1860.84, "text": " homophilic graphs where you just you can use just much simpler baselines the"}, {"start": 1860.84, "end": 1866.0, "text": " graph neural networks and so this paper just showed that so homophilic graphs are"}, {"start": 1866.0, "end": 1870.7199999999998, "text": " those graphs where the node where the nodes that are connected have similar"}, {"start": 1870.7199999999998, "end": 1876.52, "text": " label so that that's a common thing you you have in social network types of"}, {"start": 1876.52, "end": 1881.0, "text": " graphs because usually people had to have similar opinions tend to cluster"}, {"start": 1881.0, "end": 1886.24, "text": " together and those types of graphs are called homophilic and basically what"}, {"start": 1886.24, "end": 1890.96, "text": " this paper showed is that you can use a much simpler baseline you can just leave"}, {"start": 1890.96, "end": 1897.72, "text": " out a multiple layered graph neural network and just use the aggregation on"}, {"start": 1897.72, "end": 1901.84, "text": " the k-hop neighborhood so instead of using just a single a one hop"}, {"start": 1901.84, "end": 1907.1599999999999, "text": " neighborhood so your your direct connections you can use and do the the"}, {"start": 1907.1599999999999, "end": 1912.52, "text": " MPN and the message passing scheme I told you multiple times you can instead"}, {"start": 1912.52, "end": 1916.8, "text": " do the following you can just aggregate the neighbors from the k-hop"}, {"start": 1916.8, "end": 1920.8, "text": " neighborhood where K can be maybe three or four which means you can get up to"}, {"start": 1920.8, "end": 1926.0, "text": " like four under in their indirect connections and build up and then just"}, {"start": 1926.0, "end": 1931.28, "text": " use a single just so aggregate those and then use a single MPN layer to to"}, {"start": 1931.28, "end": 1935.92, "text": " build up a nice representations which can even achieve better results in those"}, {"start": 1935.92, "end": 1940.6, "text": " complex spatial methods like graph attention network etc and they are much"}, {"start": 1940.6, "end": 1945.3999999999999, "text": " more computationally efficient because they are so much simpler now now there"}, {"start": 1945.3999999999999, "end": 1950.52, "text": " is a nice parallel I can make and I will make here between the CNNs and this"}, {"start": 1950.52, "end": 1955.92, "text": " thing here so if you know anything about CNNs you know that back in 2012 when"}, {"start": 1955.92, "end": 1960.44, "text": " Alex net was introduced and produced this thing called image net moment Alex"}, {"start": 1960.44, "end": 1967.28, "text": " net had nine I think 11 by 11 kernels whereas so that's really broad compared"}, {"start": 1967.28, "end": 1973.3200000000002, "text": " to VGG which in 2015 I think basically I figured they figure out that using 3 by"}, {"start": 1973.3200000000002, "end": 1979.46, "text": " 3 kernels in much deeper architectures is actually beneficial so you can see the"}, {"start": 1979.46, "end": 1985.1200000000001, "text": " parallel between this and the CNNs and basically with the arrival of certain"}, {"start": 1985.12, "end": 1990.56, "text": " ideas such as residual connections such as better realization we were able to"}, {"start": 1990.56, "end": 1995.6799999999998, "text": " make deeper and deeper CNNs and keep those kernel kernels really small at 3"}, {"start": 1995.6799999999998, "end": 2001.1599999999999, "text": " by 3 size and that turned out to be really good so much better than using"}, {"start": 2001.1599999999999, "end": 2005.1999999999998, "text": " large kernels and shallower networks on the other hand we can see here in graph"}, {"start": 2005.1999999999998, "end": 2010.84, "text": " ML they're using big kernels and shallow networks actually produces better"}, {"start": 2010.84, "end": 2015.08, "text": " results on certain classes of graphs not on everything sometimes you still need"}, {"start": 2015.08, "end": 2020.56, "text": " deep graph neural networks and just keep that in mind okay so those were the main"}, {"start": 2020.56, "end": 2026.6399999999999, "text": " methods and now the topics that are following are a bit more esoteric but"}, {"start": 2026.6399999999999, "end": 2032.56, "text": " none nothing less useful useful from the previous methods I just explained like"}, {"start": 2032.56, "end": 2037.3999999999999, "text": " embeddings and spectrum methods and spatial methods so DNN expressivity is"}, {"start": 2037.3999999999999, "end": 2041.96, "text": " really interesting so let me let me just explain what it is in a nutshell and"}, {"start": 2041.96, "end": 2047.32, "text": " it's actually fairly easy when you put it this way you see these two graphs here"}, {"start": 2047.32, "end": 2052.92, "text": " and you can see that they are different so how we also call that more more like"}, {"start": 2052.92, "end": 2058.76, "text": " a more formal term for that would be their non isomorphic and what what you"}, {"start": 2058.76, "end": 2063.7200000000003, "text": " ideally want to have is you want to have your graph neural network assign a"}, {"start": 2063.7200000000003, "end": 2068.8, "text": " different embedding to two different to two different graphs that makes sense to"}, {"start": 2068.8, "end": 2074.1600000000003, "text": " two non isomorphic graphs on the other hand if the graphs are isomorphic you"}, {"start": 2074.1600000000003, "end": 2080.28, "text": " want to assign them the same embedding and that's the ideal case so all of the"}, {"start": 2080.28, "end": 2086.4, "text": " methods I explained you so far like GCN graph convolutional network or get are"}, {"start": 2086.4, "end": 2091.6400000000003, "text": " actually confused by this simple example as you can see the the color histogram"}, {"start": 2091.6400000000003, "end": 2096.8, "text": " we have four reds here four reds here four blues four blues two greens and two"}, {"start": 2096.8, "end": 2101.92, "text": " greens and the way how for example graph classification works you just take all"}, {"start": 2101.92, "end": 2105.7200000000003, "text": " of these features you kind of do a mean and then you'll try to do some"}, {"start": 2105.7200000000003, "end": 2109.88, "text": " prediction and because the histograms are the same in this particular case"}, {"start": 2109.88, "end": 2114.6000000000004, "text": " that means we'll have the same embedding even though they are non isomorphic so"}, {"start": 2114.6000000000004, "end": 2119.04, "text": " that's the potential failure case and we want to avoid this and so by building"}, {"start": 2119.04, "end": 2124.44, "text": " more expressive models they'll know how to cope with cases like this one and"}, {"start": 2124.44, "end": 2128.64, "text": " that's the whole so the field of expressivity deals with this exact"}, {"start": 2128.64, "end": 2133.76, "text": " problem trying to assign the same label if the graphs are isomorphic otherwise"}, {"start": 2133.76, "end": 2139.08, "text": " assign different a different not label different embeddings okay so that's the"}, {"start": 2139.08, "end": 2145.68, "text": " whole concept and this graph isomorphism network paper first showed that by using"}, {"start": 2145.68, "end": 2150.8, "text": " that GCN and GATS and all of those have there is certain families of graphs"}, {"start": 2150.8, "end": 2156.92, "text": " which confuse which confuse those MPNN networks and later a lot of cool papers"}, {"start": 2156.92, "end": 2164.4, "text": " appeared that do some sub count that use sub structures to to to increase the"}, {"start": 2164.4, "end": 2168.0800000000004, "text": " expressivity or use multiple aggregation techniques like principal"}, {"start": 2168.0800000000004, "end": 2173.5600000000004, "text": " neighborhood aggregation paper etc but that's the basic idea I want you to take"}, {"start": 2173.5600000000004, "end": 2180.36, "text": " out of this so yeah just discriminating between the nine non isomorphic graphs"}, {"start": 2180.36, "end": 2184.6400000000003, "text": " papers on exciting gene applications feel free to go through these resources"}, {"start": 2184.6400000000003, "end": 2190.1200000000003, "text": " I already walked you through these so yeah just go at your own at your own"}, {"start": 2190.1200000000003, "end": 2194.88, "text": " pace and hopefully you'll find some interesting new papers additional papers"}, {"start": 2194.88, "end": 2201.56, "text": " here okay so handling dynamic graphs chapter okay all the methods I've"}, {"start": 2201.56, "end": 2206.04, "text": " explained so far are dealing with static graphs so what's static what's dynamic"}, {"start": 2206.04, "end": 2211.32, "text": " so dynamic graphs are those that evolve with time so either new edges or new"}, {"start": 2211.32, "end": 2216.98, "text": " nodes appear as the time progresses okay so one good example would be social"}, {"start": 2216.98, "end": 2221.4, "text": " network maybe you have Facebook or Twitter where each new tweet is a new"}, {"start": 2221.4, "end": 2226.2799999999997, "text": " node that appears at certain time point yeah the same goes with edges maybe"}, {"start": 2226.2799999999997, "end": 2230.6, "text": " somebody does a read feed and that's a new edge that appeared dynamically in"}, {"start": 2230.6, "end": 2235.2799999999997, "text": " this graph so the graph is constantly being pruned and increased so it's"}, {"start": 2235.28, "end": 2239.2400000000002, "text": " usually more it's usually always increasing but there is some pruning"}, {"start": 2239.2400000000002, "end": 2243.92, "text": " happening with somebody believes the account leads the tweet whatever for"}, {"start": 2243.92, "end": 2249.1600000000003, "text": " those types of graphs we need different methods and the two main ideas I want"}, {"start": 2249.1600000000003, "end": 2253.5600000000004, "text": " you to take out from from this section is that you need to find a way to"}, {"start": 2253.5600000000004, "end": 2258.2400000000002, "text": " represent the time which is a scalar as a vector and there are a couple of"}, {"start": 2258.2400000000002, "end": 2262.96, "text": " papers you can you can find so I've linked some papers like TGN here and"}, {"start": 2262.96, "end": 2268.28, "text": " you'll you'll see some references inside like time to back I think which shows"}, {"start": 2268.28, "end": 2274.44, "text": " one of the ways you can you can kind of so map the scalar the time into its"}, {"start": 2274.44, "end": 2279.88, "text": " vector eyes representation so that's one crucial idea that's needed and the"}, {"start": 2279.88, "end": 2284.16, "text": " second one is doing the concept of doing a temporal neighborhood so basically you"}, {"start": 2284.16, "end": 2288.04, "text": " have a single node let's say it's a tweet and what you want to do is you"}, {"start": 2288.04, "end": 2295.16, "text": " want to aggregate only those edges there may be at most one hour like old and you"}, {"start": 2295.16, "end": 2298.68, "text": " want to just ignore all of the other edges otherwise you have a huge graph"}, {"start": 2298.68, "end": 2302.04, "text": " and you have this thing called over squashing problem because you have so"}, {"start": 2302.04, "end": 2305.7599999999998, "text": " many nodes and you want to try to aggregate them usually doing some mean"}, {"start": 2305.7599999999998, "end": 2311.56, "text": " or something and that's kind of that's a similar effect happens in those bag of"}, {"start": 2311.56, "end": 2315.52, "text": " words if you're familiar with bag of words technique so you kind of know that"}, {"start": 2315.52, "end": 2322.28, "text": " by doing bag of roots and you lose the information same thing happens in RNNs"}, {"start": 2322.28, "end": 2327.12, "text": " where you get these this over squashing effect because as the sequence becomes"}, {"start": 2327.12, "end": 2331.64, "text": " longer you're kind of doing this bag of word thing and you're losing the older"}, {"start": 2331.64, "end": 2336.36, "text": " information so what so transformers for example tackled this problem by using"}, {"start": 2336.36, "end": 2341.88, "text": " the concept of attention right so you give more attention to more important"}, {"start": 2341.88, "end": 2348.7200000000003, "text": " feature vectors and ignore some less important ones so yeah basically temporal"}, {"start": 2348.7200000000003, "end": 2352.44, "text": " neighborhood helps us avoid that over squashing problems problem that happens"}, {"start": 2352.44, "end": 2358.4, "text": " in all of those bag of word methods such as in RNNs I linked a couple of"}, {"start": 2358.4, "end": 2365.48, "text": " unsupervised papers unsupervised learning papers like DJI and this"}, {"start": 2365.48, "end": 2370.6400000000003, "text": " variational graph autoencoder so if you're not familiar with variational"}, {"start": 2370.64, "end": 2374.3199999999997, "text": " autoencoder this paper probably won't make any sense it's just it gets three"}, {"start": 2374.3199999999997, "end": 2378.3599999999997, "text": " or four pages it's three or four pages long and it already assumes you know"}, {"start": 2378.3599999999997, "end": 2382.8799999999997, "text": " everything about variational autoencoders but it's a it's a nice read"}, {"start": 2382.8799999999997, "end": 2387.24, "text": " if you're interested in learning more about these unsupervised learning methods"}, {"start": 2387.24, "end": 2392.8399999999997, "text": " just make it I made a small tangent here on the this field called geometric deep"}, {"start": 2392.8399999999997, "end": 2397.0, "text": " learning so when you start doing graph ML you'll hear about geometric deep"}, {"start": 2397.0, "end": 2402.48, "text": " learning that's for sure and those are pretty much separate it's a pretty much"}, {"start": 2402.48, "end": 2407.52, "text": " separate research area which deals with manifolds and not with graphs per se so"}, {"start": 2407.52, "end": 2412.56, "text": " even though again you can treat certain manifolds once you discretize them them"}, {"start": 2412.56, "end": 2417.44, "text": " there they are basically graphs but here the same thing as with computer vision"}, {"start": 2417.44, "end": 2421.96, "text": " the same thing as with NLP you have a class of algorithms which are much"}, {"start": 2421.96, "end": 2426.8, "text": " better optimized to deal with manifolds even though they can be treated when"}, {"start": 2426.8, "end": 2432.32, "text": " discretized as graphs so just keep that in mind and don't make it too big of a"}, {"start": 2432.32, "end": 2436.76, "text": " detour into that field because it's pretty much separate research area"}, {"start": 2436.76, "end": 2443.0, "text": " compared to graph ML okay with that out of the way I linked a couple of cool"}, {"start": 2443.0, "end": 2447.6400000000003, "text": " visualization tools here like iGraph I've used iGraph personally in my graph"}, {"start": 2447.6400000000003, "end": 2452.0800000000004, "text": " attention network project check that one out if you haven't I'll link it down in"}, {"start": 2452.08, "end": 2460.56, "text": " the description as well and aside from that I've been mentioning data sets"}, {"start": 2460.56, "end": 2466.24, "text": " here and data sets are so important and they usually achieve not they usually"}, {"start": 2466.24, "end": 2471.04, "text": " don't get so much attention in the community usually you have something"}, {"start": 2471.04, "end": 2475.96, "text": " like ah there's this huge model like GPT that has like five through trillion"}, {"start": 2475.96, "end": 2481.6, "text": " parameters and that gets really hyped up but the data sets which are so"}, {"start": 2481.6, "end": 2487.0, "text": " important for progressing the research don't get so much attention and there"}, {"start": 2487.0, "end": 2492.96, "text": " is a problem with graph ML that people use these Cora sites here PPI and PubMed"}, {"start": 2492.96, "end": 2499.2799999999997, "text": " data sets a lot in the over the previous years and only recently have we started"}, {"start": 2499.2799999999997, "end": 2504.3199999999997, "text": " to integrate bigger real-world graphs and use them as benchmarks so what"}, {"start": 2504.3199999999997, "end": 2509.72, "text": " happened is that when you use these these let's call them toy benchmark"}, {"start": 2509.72, "end": 2515.24, "text": " graph data says like similar to amnesty computer vision you basically can't"}, {"start": 2515.24, "end": 2520.8799999999997, "text": " discern between much more expressive and better models from simpler baselines so"}, {"start": 2520.8799999999997, "end": 2523.8799999999997, "text": " there's the whole point of the benchmark right you want to you want to have a"}, {"start": 2523.8799999999997, "end": 2529.48, "text": " benchmark which clearly separates and fill sorts better algorithms from the"}, {"start": 2529.48, "end": 2534.8799999999997, "text": " worst ones and here the the the margin became smaller and smaller even between"}, {"start": 2534.8799999999997, "end": 2539.56, "text": " the simplest baselines and the most complex GNNs so we needed new data sets"}, {"start": 2539.56, "end": 2544.04, "text": " and so this I link this open graph benchmark which is a nice initiative"}, {"start": 2544.04, "end": 2551.4, "text": " that introduced much better much much larger and more diverse graphs into"}, {"start": 2551.4, "end": 2556.7999999999997, "text": " this graph ML field and that's a really good thing and we should be very"}, {"start": 2556.7999999999997, "end": 2561.88, "text": " grateful for people who invest their time to develop better benchmarks even"}, {"start": 2561.88, "end": 2566.36, "text": " though that's not as exciting as developing some proposing some novel"}, {"start": 2566.36, "end": 2571.76, "text": " novel idea novel it's usually research is usually super incremental rarely do"}, {"start": 2571.76, "end": 2575.8, "text": " you see something that's completely new like graph embedding methods Vertweck"}, {"start": 2575.8, "end": 2581.4, "text": " already existed they just adapted it to to graphs similarly for most of these"}, {"start": 2581.4, "end": 2587.48, "text": " methods are just a variation on some research that already existed just keep"}, {"start": 2587.48, "end": 2592.36, "text": " that in mind I linked some cool resources Michael Bronstein's got a"}, {"start": 2592.36, "end": 2598.96, "text": " really awesome blog highly recommend you go and read those but probably you'll"}, {"start": 2598.96, "end": 2602.7200000000003, "text": " want to leave those for a bit later because he's a professor and he has a"}, {"start": 2602.7200000000003, "end": 2607.6, "text": " that his blogs are a bit more technical so you want to start with those high"}, {"start": 2607.6, "end": 2612.1600000000003, "text": " level blogs read through resources I've listed and only then start reading"}, {"start": 2612.1600000000003, "end": 2618.96, "text": " Michael's blog blog finally once you get some once you get your foot inside"}, {"start": 2618.96, "end": 2623.64, "text": " graph ML start following people in Twitter and start following Sergey's"}, {"start": 2623.64, "end": 2628.68, "text": " newsletter I highly recommend it that's a good way to keep in sync with the"}, {"start": 2628.68, "end": 2633.16, "text": " newest things happening in a specific field so similarly to how Sergey has"}, {"start": 2633.16, "end": 2640.2400000000002, "text": " newsletter in graph ML Sebastian Huda has a really nice newsletter in the world of"}, {"start": 2640.2400000000002, "end": 2646.04, "text": " natural language processing so yeah this is something that was shown to be a nice"}, {"start": 2646.04, "end": 2650.32, "text": " format for transferring knowledge in the community so you should be following"}, {"start": 2650.32, "end": 2655.7599999999998, "text": " these small specialized newsletters highly recommended I'll skip through this"}, {"start": 2655.7599999999998, "end": 2661.72, "text": " I had I made a couple of videos on graph attention network project they did so"}, {"start": 2661.72, "end": 2667.84, "text": " we gained a lot of traction like it got 750 stars since I published it a couple"}, {"start": 2667.84, "end": 2672.56, "text": " of weeks ago and you can go through it you'll find it useful probably I've got"}, {"start": 2672.56, "end": 2676.4, "text": " both the trans active as well as the inductive examples inside like both the"}, {"start": 2676.4, "end": 2680.52, "text": " Cora which is trans active and PPI protein protein interaction data set"}, {"start": 2680.52, "end": 2686.56, "text": " which is inductive and I've got the I made this Jupiter notebook as well so"}, {"start": 2686.56, "end": 2692.36, "text": " yeah do check it out that's the project and that's pretty much everything I"}, {"start": 2692.36, "end": 2697.2, "text": " wanted to tell you maybe a short note on some other detours you could be making"}, {"start": 2697.2, "end": 2701.52, "text": " and that's I already mentioned geometric deep learning so it's a in a way it's a"}, {"start": 2701.52, "end": 2706.4, "text": " separate research topic so be aware don't don't just don't make a mistake and"}, {"start": 2706.4, "end": 2711.72, "text": " start reading those papers because because they are pretty much independent"}, {"start": 2711.72, "end": 2716.0, "text": " orthogonal to the graph ML papers then there is this equivalent deep learning"}, {"start": 2716.0, "end": 2721.16, "text": " subfield which deals with making your algorithms much more statistically"}, {"start": 2721.16, "end": 2726.92, "text": " efficient by exploiting symmetries of your problem so the same way as CNN's"}, {"start": 2726.92, "end": 2733.48, "text": " are translationally equivalent they are trying to make models which are also"}, {"start": 2733.48, "end": 2738.56, "text": " rotationally equivalent which can just exploit different symmetries and again"}, {"start": 2738.56, "end": 2744.48, "text": " it's a separate research subfield be aware of it don't don't waste too much"}, {"start": 2744.48, "end": 2750.4, "text": " time of going there if you just want to focus on graph ML even though again"}, {"start": 2750.4, "end": 2754.88, "text": " there is always some cross-semination of ideas so you have to draw the line where"}, {"start": 2754.88, "end": 2761.36, "text": " you want to explore more versus exploit more finally knowledge graphs there's"}, {"start": 2761.36, "end": 2766.92, "text": " something there are mix between NLP and graph ML so again totally new"}, {"start": 2766.92, "end": 2773.48, "text": " terminology totally new set of people history papers so just be cognizant of"}, {"start": 2773.48, "end": 2777.44, "text": " that fact that's it you know the drill hit that subscribe button hit the bell"}, {"start": 2777.44, "end": 2791.08, "text": " icon and until next time keep learning deep"}]