Graph Neural Network Project Update! (I'm coding GAT from scratch)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

A short update on some of my previous projects as well as my current project: the Graph Attention Network (GAT).

You'll learn about:
✔️My previous deep learning projects
✔️Some of the challenges I faced implementing GAT

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Neural Style Transfer (original): https://github.com/gordicaleksa/pytorch-neural-style-transfer
✅ Neural Style Transfer (fast): https://github.com/gordicaleksa/pytorch-neural-style-transfer-fast
✅ NST + video segmentation: https://github.com/gordicaleksa/pytorch-naive-video-neural-style-transfer
✅ DeepDream: https://github.com/gordicaleksa/pytorch-deepdream
✅ GANs: https://github.com/gordicaleksa/pytorch-gans
✅ Vaswani's transformer: https://github.com/gordicaleksa/pytorch-original-transformer

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

0:00 My previous deep learning projects
2:10 GAT
3:20 Profiling three/four different GAT implementations
5:10 Profiling different sparse formats (COO, LIL, CSR, ...)
6:10 t-SNE, UMAP
6:30 Patreon shoutout
6:45 Love you folks

Credits: some of the images I used belong to other creators.
I created this video for purely educational purposes without the goal of monetizing it.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#graphattentionnetwork #projectupdate #deeplearning
 What's up folks? So this video is going to be an update and not a classical video on my channel where I'm just covering some research paper. So basically what I've been doing is over the last seven days I think I've been coding this graph attention network or GATT for short from scratch and I actually started somewhere around New Year's time and I just continued now. I'll be just skipping the the classical research paper overview because for those of you who know me you know that I do I mix a lot of theoretical work but I also do a lot of hands-on like deep learning projects and stuff. So now the time has come pretty much in my like workflow to just kind of pause the theory a little bit and focus on coding. As you probably know like my github profile has a lot of projects those are some of the the projects I've been developing throughout the last year. Make sure to check them out I've got a neural style transfer on my github you can play with that. I've implemented both Gatiss's approach so that's the original optimization based approach. I've also got a Johnson's approach which is the fast-forward like just using CNN to stylize your images. I've got us even some really cool projects where you can just basically segment yourself and stylize everything around or stylize yourself whatever. So that's cool check it out as well. Then I've got deep dream which was really awesome artistic project and you can learn a lot about CNNs and deep learning in general by doing this project so make sure to check that one as well. Then I've been developing generative adversarial networks as well so I did like develop the original one I did develop the conditional GAN and finally the DC GAN which kind of started the Cambrian explosion in the world of generative adversarial networks so that one is really cool. Then continuing I was developing the transformer I was working on transformers and I actually open-sourced the original was finally transformer so I really like to think that that project is a super super good place to add to start supplementing the paper and understanding how it actually works so every single detail so we got it's got a lot of comments so yeah hopefully you'll find that one useful as well. So now I'm working on GAT graph attention at work and as any as pretty much any deep learning project you can conceptually I kind of split it into a couple parts like the first one is data then we have the the model part and finally we have the training loop and some additional visualizations the stuff I usually put in playground a cool playground whatever and basically when it comes to data because these are graphs I've been doing some stuff that they usually don't do when I'm exploring data so I've been doing network analysis like just plotting the degree distribution of my network of core network and some other statistics I did some I found this really good tool iGraph and I actually plotted how core looks like and that kind of just gives me a gut feeling for for how the graph looks like and that's that's kind of useful you feel better about yourself nothing else you probably don't need to know all of those details to make a to make a this project successful but it's it's really cool and you learn a lot in the process to learn different tools you learn yeah basically that and finally the the model part is so I have like four I think I have four different implementations so the first one is similar to pie tour geometric if you're not familiar with that project that's a really super awesome project that was implementing most of the main graph networks so GNN so do check it out but the thing is my implementation is actually like almost twice as fast because I was using some specialized pie torch functions like instead of using basic non-py indexing I was using index select I was using instead of using like some suffix function I was using torch some I also don't have the overhead of having to fit my my get project inside of this message passing framework as they do so it's it's it's cleaner and it's faster so but it's only for graph attention network so hopefully you'll find it useful I'll open source it I think next weekend so stay tuned for their one and the other implementations one is using towards sparse API which is currently in beta so that's going to be fun because I'll have to develop the back prop function explicitly so it doesn't handle so those sparse functions don't handle the back prop automatically the third one is the conceptually the the easiest to understand but it's computationally most inefficient so yeah so that's the gradation the training loop part is really easy because core is just a simple classification problem so nothing new there just cross entropy like some masking because we have because it's a transactive training setting and that means you you have to mask out the training nodes but you can see the test nodes as well as the validation nodes during the procedure so yeah that's nothing interesting there then I've got the playground where I've been profiling different sparse formats for different matrices so like you probably heard of some of them like COO the coordinate form format then we have D okay that we have little we have CSR CSS see as no CSC sorry then we've got not CSS that's that's that's not the best format and then we've got Daya and BSR so about about seven formats which scipy supports and what I've did is I just analyzed how fast they are and actually figure out that some of the implementations out there like GCN by Thomas Kupf and even get the official get by Petar Velichkovic used they've used a little format for some arithmetic operations like summations and stuff like that and that's actually suboptimal because you want to use that one as you're building up your graph as you're just modifying the sparsity structure but you don't want to use it for arithmetic operations so yeah that's a that's a fun fun it was a fun experimentation and also of course I love visualizing stuff I played a little bit with t-sne where I visualized the embeddings that come out from the train get model so yeah I also play with UMAP that's a that's a relatively new algorithm most of the people are just using t-sne as the default method but UMAP is also cool but in this particular case I actually found that t-sne is as good as UMAP so I just kind of dumped it all together and finally I want to make a shout out to my patrons to Petar Velichkovic from DeepMind who is the AI overlord of this channel and to Jimmix who is the AI disciple of this channel nothing less important just wanted to make a shout out and thank you for for supporting my channel nothing less important the whole community I love all of your comments keep them going I just want to read a couple of them here so the first one great one question have you noticed that arbitrary style transfer paper has more citations than universal style transfer one so I love those geeky ones as well as these more simple ones like geez man do you ever take a holiday lol I appreciate all your videos so just keep them coming I love your comments and I read all of your comments and I'm gonna try and answer all of them hi Alexa can you make a video about your background and your journey into AI ML yeah basically if I see more of these comments I'll probably make some time and create a video where I'll walk you through my journey into deep learning so also want to make a shout out to to Phil who is also supporting my channel and thank you your channel is also amazing so love to collaborate one day with you yeah that was pretty much it you know the drill subscribe hit the like button and share this video that's how you can help me build up this channel build up the community I want to include you as more as I can in in my in all of this so if you have any suggestions how I can do that please write them down in the comment section and until next time keep learning deep
[{"start": 0.0, "end": 5.32, "text": " What's up folks? So this video is going to be an update and not a classical video"}, {"start": 5.32, "end": 10.36, "text": " on my channel where I'm just covering some research paper. So basically what"}, {"start": 10.36, "end": 15.3, "text": " I've been doing is over the last seven days I think I've been coding this graph"}, {"start": 15.3, "end": 19.16, "text": " attention network or GATT for short from scratch and I actually started"}, {"start": 19.16, "end": 23.080000000000002, "text": " somewhere around New Year's time and I just continued now. I'll be just skipping"}, {"start": 23.080000000000002, "end": 28.96, "text": " the the classical research paper overview because for those of you who"}, {"start": 28.96, "end": 33.64, "text": " know me you know that I do I mix a lot of theoretical work but I also do a lot"}, {"start": 33.64, "end": 38.760000000000005, "text": " of hands-on like deep learning projects and stuff. So now the time has come"}, {"start": 38.760000000000005, "end": 43.88, "text": " pretty much in my like workflow to just kind of pause the theory a"}, {"start": 43.88, "end": 49.28, "text": " little bit and focus on coding. As you probably know like my github profile"}, {"start": 49.28, "end": 52.88, "text": " has a lot of projects those are some of the the projects I've been developing"}, {"start": 52.88, "end": 56.6, "text": " throughout the last year. Make sure to check them out I've got a neural style"}, {"start": 56.6, "end": 60.6, "text": " transfer on my github you can play with that. I've implemented both Gatiss's"}, {"start": 60.6, "end": 63.800000000000004, "text": " approach so that's the original optimization based approach. I've also"}, {"start": 63.800000000000004, "end": 68.68, "text": " got a Johnson's approach which is the fast-forward like just using CNN to"}, {"start": 68.68, "end": 74.64, "text": " stylize your images. I've got us even some really cool projects where you can"}, {"start": 74.64, "end": 78.84, "text": " just basically segment yourself and stylize everything around or stylize"}, {"start": 78.84, "end": 83.12, "text": " yourself whatever. So that's cool check it out as well. Then I've got deep dream"}, {"start": 83.12, "end": 86.72, "text": " which was really awesome artistic project and you can learn a lot about"}, {"start": 86.72, "end": 90.68, "text": " CNNs and deep learning in general by doing this project so make sure to check"}, {"start": 90.68, "end": 94.64, "text": " that one as well. Then I've been developing generative adversarial"}, {"start": 94.64, "end": 98.88000000000001, "text": " networks as well so I did like develop the original one I did develop the"}, {"start": 98.88000000000001, "end": 103.64, "text": " conditional GAN and finally the DC GAN which kind of started the Cambrian"}, {"start": 103.64, "end": 107.52000000000001, "text": " explosion in the world of generative adversarial networks so that one is"}, {"start": 107.52000000000001, "end": 111.56, "text": " really cool. Then continuing I was developing the transformer I was working"}, {"start": 111.56, "end": 114.96000000000001, "text": " on transformers and I actually open-sourced the original was finally"}, {"start": 114.96000000000001, "end": 120.48, "text": " transformer so I really like to think that that project is a super super good"}, {"start": 120.48, "end": 125.16, "text": " place to add to start supplementing the paper and understanding how it actually"}, {"start": 125.16, "end": 129.68, "text": " works so every single detail so we got it's got a lot of comments so yeah"}, {"start": 129.68, "end": 134.16, "text": " hopefully you'll find that one useful as well. So now I'm working on GAT graph"}, {"start": 134.16, "end": 139.48000000000002, "text": " attention at work and as any as pretty much any deep learning project you can"}, {"start": 139.48, "end": 143.88, "text": " conceptually I kind of split it into a couple parts like the first one is data"}, {"start": 143.88, "end": 148.0, "text": " then we have the the model part and finally we have the training loop and"}, {"start": 148.0, "end": 152.39999999999998, "text": " some additional visualizations the stuff I usually put in playground a cool"}, {"start": 152.39999999999998, "end": 158.04, "text": " playground whatever and basically when it comes to data because these are"}, {"start": 158.04, "end": 162.51999999999998, "text": " graphs I've been doing some stuff that they usually don't do when I'm exploring"}, {"start": 162.51999999999998, "end": 166.67999999999998, "text": " data so I've been doing network analysis like just plotting the degree"}, {"start": 166.68, "end": 171.52, "text": " distribution of my network of core network and some other statistics I did"}, {"start": 171.52, "end": 176.92000000000002, "text": " some I found this really good tool iGraph and I actually plotted how core"}, {"start": 176.92000000000002, "end": 180.56, "text": " looks like and that kind of just gives me a gut feeling for for how the graph"}, {"start": 180.56, "end": 184.12, "text": " looks like and that's that's kind of useful you feel better about yourself"}, {"start": 184.12, "end": 188.68, "text": " nothing else you probably don't need to know all of those details to make a to"}, {"start": 188.68, "end": 192.68, "text": " make a this project successful but it's it's really cool and you learn a lot in"}, {"start": 192.68, "end": 198.64000000000001, "text": " the process to learn different tools you learn yeah basically that and finally"}, {"start": 198.64000000000001, "end": 203.96, "text": " the the model part is so I have like four I think I have four different"}, {"start": 203.96, "end": 208.12, "text": " implementations so the first one is similar to pie tour geometric if you're"}, {"start": 208.12, "end": 212.24, "text": " not familiar with that project that's a really super awesome project that was"}, {"start": 212.24, "end": 219.24, "text": " implementing most of the main graph networks so GNN so do check it out but"}, {"start": 219.24, "end": 222.68, "text": " the thing is my implementation is actually like almost twice as fast"}, {"start": 222.68, "end": 226.64000000000001, "text": " because I was using some specialized pie torch functions like instead of using"}, {"start": 226.64000000000001, "end": 231.44, "text": " basic non-py indexing I was using index select I was using instead of using like"}, {"start": 231.44, "end": 237.64000000000001, "text": " some suffix function I was using torch some I also don't have the overhead of"}, {"start": 237.64000000000001, "end": 242.36, "text": " having to fit my my get project inside of this message passing framework as"}, {"start": 242.36, "end": 248.44, "text": " they do so it's it's it's cleaner and it's faster so but it's only for graph"}, {"start": 248.44, "end": 252.2, "text": " attention network so hopefully you'll find it useful I'll open source it I"}, {"start": 252.2, "end": 257.8, "text": " think next weekend so stay tuned for their one and the other implementations"}, {"start": 257.8, "end": 264.0, "text": " one is using towards sparse API which is currently in beta so that's going to be"}, {"start": 264.0, "end": 268.15999999999997, "text": " fun because I'll have to develop the back prop function explicitly so it"}, {"start": 268.15999999999997, "end": 272.04, "text": " doesn't handle so those sparse functions don't handle the back prop"}, {"start": 272.04, "end": 276.48, "text": " automatically the third one is the conceptually the the easiest to"}, {"start": 276.48, "end": 280.48, "text": " understand but it's computationally most inefficient so yeah so that's the"}, {"start": 280.48, "end": 284.22, "text": " gradation the training loop part is really easy because core is just a"}, {"start": 284.22, "end": 288.36, "text": " simple classification problem so nothing new there just cross entropy like some"}, {"start": 288.36, "end": 293.24, "text": " masking because we have because it's a transactive training setting and that"}, {"start": 293.24, "end": 300.0, "text": " means you you have to mask out the training nodes but you can see the test"}, {"start": 300.0, "end": 305.32, "text": " nodes as well as the validation nodes during the procedure so yeah that's"}, {"start": 305.32, "end": 309.76, "text": " nothing interesting there then I've got the playground where I've been profiling"}, {"start": 309.76, "end": 315.2, "text": " different sparse formats for different matrices so like you probably heard of"}, {"start": 315.2, "end": 320.56, "text": " some of them like COO the coordinate form format then we have D okay that we"}, {"start": 320.56, "end": 328.52, "text": " have little we have CSR CSS see as no CSC sorry then we've got not CSS that's"}, {"start": 328.52, "end": 335.0, "text": " that's that's not the best format and then we've got Daya and BSR so about"}, {"start": 335.0, "end": 340.4, "text": " about seven formats which scipy supports and what I've did is I just analyzed how"}, {"start": 340.4, "end": 344.16, "text": " fast they are and actually figure out that some of the implementations out"}, {"start": 344.16, "end": 347.84, "text": " there like GCN by Thomas Kupf and even get the official get by Petar"}, {"start": 347.84, "end": 353.76, "text": " Velichkovic used they've used a little format for some arithmetic operations"}, {"start": 353.76, "end": 358.24, "text": " like summations and stuff like that and that's actually suboptimal because you"}, {"start": 358.24, "end": 361.36, "text": " want to use that one as you're building up your graph as you're just modifying"}, {"start": 361.36, "end": 365.92, "text": " the sparsity structure but you don't want to use it for arithmetic operations"}, {"start": 365.92, "end": 371.48, "text": " so yeah that's a that's a fun fun it was a fun experimentation and also of course"}, {"start": 371.48, "end": 375.52000000000004, "text": " I love visualizing stuff I played a little bit with t-sne where I visualized"}, {"start": 375.52000000000004, "end": 380.52000000000004, "text": " the embeddings that come out from the train get model so yeah I also play with"}, {"start": 380.52000000000004, "end": 384.56, "text": " UMAP that's a that's a relatively new algorithm most of the people are just"}, {"start": 384.56, "end": 388.40000000000003, "text": " using t-sne as the default method but UMAP is also cool but in this particular"}, {"start": 388.4, "end": 392.96, "text": " case I actually found that t-sne is as good as UMAP so I just kind of dumped it"}, {"start": 392.96, "end": 396.79999999999995, "text": " all together and finally I want to make a shout out to my patrons to Petar"}, {"start": 396.79999999999995, "end": 401.35999999999996, "text": " Velichkovic from DeepMind who is the AI overlord of this channel and to"}, {"start": 401.35999999999996, "end": 405.15999999999997, "text": " Jimmix who is the AI disciple of this channel nothing less important just"}, {"start": 405.15999999999997, "end": 408.85999999999996, "text": " wanted to make a shout out and thank you for for supporting my channel nothing"}, {"start": 408.85999999999996, "end": 413.12, "text": " less important the whole community I love all of your comments keep them"}, {"start": 413.12, "end": 417.79999999999995, "text": " going I just want to read a couple of them here so the first one great one"}, {"start": 417.8, "end": 421.48, "text": " question have you noticed that arbitrary style transfer paper has more citations"}, {"start": 421.48, "end": 426.64, "text": " than universal style transfer one so I love those geeky ones as well as these"}, {"start": 426.64, "end": 431.64, "text": " more simple ones like geez man do you ever take a holiday lol I appreciate"}, {"start": 431.64, "end": 436.24, "text": " all your videos so just keep them coming I love your comments and I read all of"}, {"start": 436.24, "end": 440.52, "text": " your comments and I'm gonna try and answer all of them hi Alexa can you make"}, {"start": 440.52, "end": 445.52, "text": " a video about your background and your journey into AI ML yeah basically if I"}, {"start": 445.52, "end": 450.84, "text": " see more of these comments I'll probably make some time and create a video where"}, {"start": 450.84, "end": 454.64, "text": " I'll walk you through my journey into deep learning so also want to make a"}, {"start": 454.64, "end": 459.56, "text": " shout out to to Phil who is also supporting my channel and thank you your"}, {"start": 459.56, "end": 464.71999999999997, "text": " channel is also amazing so love to collaborate one day with you yeah that"}, {"start": 464.71999999999997, "end": 469.08, "text": " was pretty much it you know the drill subscribe hit the like button and share"}, {"start": 469.08, "end": 472.28, "text": " this video that's how you can help me build up this channel build up the"}, {"start": 472.28, "end": 477.4, "text": " community I want to include you as more as I can in in my in all of this so if"}, {"start": 477.4, "end": 480.55999999999995, "text": " you have any suggestions how I can do that please write them down in the"}, {"start": 480.56, "end": 505.16, "text": " comment section and until next time keep learning deep"}]