How to learn deep learning? (Transformers Example)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I walk you through how I approach tackling a new deep learning topic and how my learning program looks like!

You'll learn about:
✔️ My strategy for learning ANY new (deep learning) topic fast
✔️ Lots of learning/personal tips along the way

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ https://jalammar.github.io/illustrated-transformer/ 

✅ Link to pdf of my OneNote page: https://1drv.ms/b/s!AiUr5QUl32i_0B54CcWPM7JcLDDN?e=p8QoEL
Note: it's a snapshot from November 12th, 2020.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
0:00 Quick recap of my 2020 deep learning journey
0:55 Tricks I learned doing my past projects
4:11 What I learned from researching NST
6:30 Deep Dream project
8:25 GANs project
10:00 Going forward - transformers!
10:36 Why transformers?
12:47 OneNote walk-through (attention mechanism)
15:30 OneNote (self-attention mechanism)
17:40 Zoom out - is there a life after GPT?
18:50 Word embeddings (word to vec, GloVe)
20:30 Tokenization - the underdog of NLP
22:00 Main BERT dependencies
25:30 Going forward 
27:00 Always keep the bigger picture in your head

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#deeplearning #deeplearningtips #learningtips
 What's up folks? So in this video I had this idea of sharing with you how I approach learning a new deep learning topic. So over the past seven months I've been learning about neural style transfer for maybe three, four months. That took a while because I really wanted to get a familiarity with PyTorch and yeah, basically PyTorch. Then I did some Deep Dream project and that took maybe a month and a half and my last big deep learning topic exploration was Verganz and I did those for maybe two months. Open sourced a project there and I'm currently doing Transformers and I'll be walking you through my OneNote and how I accumulate my notes and how I approach organizing information and trying to get better at whatever topic I choose to study at the present. So as I said I basically started with neural style transfer earlier this year and what I did was I first started reading lots of research papers and I do understand that if you're a beginner you won't start with research papers but there are lots of blogs and some high-level resources which you could leverage to get a better understanding of the field and so I can show you here I basically yeah I just started with the original neural style transfer paper and that's this one by Gatiss and so basically while reading the paper you you find some references and reading blogs you figure out there are some better methods now and then and that's how you start slowly accumulating good research papers to read and also like using simply using Google and figuring out how many citations a paper has that's also one of the probably good indicators of whether you should be reading it or not. So as you can see here so I've been reading like I read maybe 18 papers on neural style transfer and also I don't I don't read every single paper thoroughly some of those I just kind of glimpse and so I organized it like this so I ran through those whatever like silly name just a directory where I dump the papers which I just kind of have some overview but I am not thoroughly familiar with those and these are kind of some papers which are on my backlog and I don't think I'll be I'll have time to to go through these but yeah they are they are still here anyways so after running through the theory what I did was basically started implementing stuff from scratch and that's super important so theory must be complemented with code otherwise it will be it won't be the same you you won't have a deep understanding of how stuff works because devil is in the details like the first paper I started implementing was Gadeus's original paper and I thought I knew everything works but I didn't turned out I didn't and so I struggle with some details like should you use 0 to 25 to 55 range or 0 to 1 range and there's so many of those small details which you don't think about when you're just reading theory so what I did was I implemented the original paper here then I I implemented some Johnson's paper which didn't use the optimization method they just use convolutional neural network to you basically input an image it passes through the convolutional neural net and out goes the stylized version of that same image and finally I did play with videos so I did implement like a naive approach of neural style transfer for videos naive in the sense that I don't have any temporal constraints included so it's kind of jittery when you go from frame to frame but I did play with some like models which actually during the training include the temporal loss and so the result is much more stable once you start doing the inference let's go and take a look at a couple of the projects implemented so the first one was so the optimization method by Gadeus and you'll look the original neural style transfer algorithm and as you can see here I do think I'm a big believer that you should create a really good readme file otherwise people will get confused nobody wants to have to go deep inside the code you know to understand what's happening you should present what your what your code is doing so that it's easy for the others to to know straight ahead whether they want to play with your code or not and you should put some basic instructions how to set up your code how to use your code and maybe how to debug and experiment and whatever additional things you have and also like putting acknowledgments is really nice because you always build up on somebody else's work or you use it as an inspiration to build your own work and many do not admit this but I think it's like everybody knows that everybody's using other people's code and either building off of it or using it as an inspiration so that was the that was the first project I did and then the second one as I mentioned was Johnson's same similar thing it's this one is much faster it can run almost real-time but it's has a bit worse quality than than the Gattas project and finally this is the video project I was talking about so what I did here additionally was I included a small part that will actually segment the the the user that's in the frame so so so it's basically built for a specific setup and that's a person talking in front of the camera like I'm doing right now and it does a pretty decent job in that setup as you can see here so what you can do after you find the segmentation mask is you can apply different styles to either background or foreground and that's that's a useful useful nice thing to have and here is a small GIF I created using that project so you can go ahead and play with these so the main idea here is just to show you how how I personally work and maybe you'll find some inspiration hopefully do if you do please leave a comment whether this video helped you or did not next thing I did as I mentioned was deep dream and now you may wonder why is this guy doing this nice stylistic projects I mean you don't learn anything pragmatic right so the thing is in Microsoft I'm doing lots of really practical stuff and so I have this luxury of just playing around with the different techniques different research areas in my free time so the thing is you learn a lot doing these projects so for example doing neural style transfer you you'll learn about the concept called perceptual loss for example that's used all around the place in other areas also for example the Johnson the paper I mentioned with the like the Johnson's paper he the architecture he used for the generation of neural style transfer images is something that was was being really used in and I've seen it reused in bunch of other papers so you do get to extract some useful information from one field and transfer it to another so one additionally so there's this thing called adaptive instant normalization and it was invented in the context of the neural style transfer field but I recently saw it while researching GANs and the famous style again architecture is actually using adaptive instance normalization so that's that's the cross-semination I was talking about so basically do learn a lot by doing some maybe unrelated field but it is related to whatever you'll be doing in the future if you if you decide to keep doing deep learning anyways this is how deep dream algorithm looks like I think it's got really decent images just super exciting to see that the neural network itself has this kind of power inside and that you can achieve this by just using simple optimization methods and CNNs and I did create some gifts here and that's all awesome jumping into GANs quickly so that was my final deep learning topic I researched and again read bunch of papers that's usually how I start exploring the field actually I'm lying I actually start usually start with high-level resources but I'll get to details a bit later with transformers so after reading papers I just developed from scratch like the vanilla the original generative adversarial network then I did some conditional GANs where you can use you can condition which class you want to generate so in particular I did amnest and I used amnest data here so what I achieved was that I can basically choose which class I want to generate so here you can see I can condition the generator to create only zeros or only ones etc and finally I developed this DC GAN paper which was super popular and it's super famous because it basically created this Cambrian explosion of new research papers that were using DC GAN as a building block and it really accelerated the field as such so like doing doing this I learned a lot and that's something I recommend to everybody that's that's a really good way to learn start with theory don't dwell too much on theory maybe two weeks like tops and then start start coding it up from scratch I think I mentioned this I still didn't so I'm currently doing transformers I'm in this phase of exploring the theory that's behind transformers so I still don't have as you can see here I don't have the project implemented yet but I'll start doing it like this Monday I think so hopefully you folks will be able to play with the code once I finish it up in maybe two weeks time so let me let me walk you through my one note and show you how I approach learning a new deep learning fields I don't start with research papers usually unless I'm really familiar with the field I do start with some high-level resources and let's just check out how it looks like so if I open my one note you can see a bunch of like resources I've accumulated over the last two weeks actually because I've been doing this pretty much every single day and yeah let me start slowly digesting this for you so first a little bit of context how I started and why I started doing transformers so first of all you all heard of GPT-3 and the super famous language model by OpenAI and the cool stuff it's been creating over the last couple of months and lots of hype around there so that wasn't actually the reason I started doing transformers even though they were on my backlog I would I would have started learning transformers at one point of time but it's just kind of happened sooner because what happened is this visual transformer and it was published I think around 3rd of October on this machine learning conference called iClear I think that's how it's pronounced and basically what it did it showed that transformers are more powerful than CNNs in the field of computer vision in specifically in the big data regime and it kind of blew my mind off and I was like I have to start doing this sooner so there was a short background story why I started doing transformers and what I first did was I just opened the paper visual transformer and I wasn't an NLP guy I was a computer vision guy so I've been spending most of my time over the last couple of years doing computer vision and I started reading it and I basically immediately saw it has a really strong dependency on the original transformer paper so that's what I did I took a step back I opened the transformer paper and I've read it from top to bottom and basically after finishing the paper then I read the visual transformer and I still didn't have a clue of what's happening many details I mean I did have a rough understanding what's happening but I didn't have many details so I decided I have to take a step even one step back and start learning about the attention mechanism and some other dependencies that those papers had like like word embeddings and concepts like like that one let's finally start walking through the the one note so as you can see here that this section is attention as I mentioned I figure out I'll need to understand the tension mechanism thoroughly so what I'd start what I started doing was first just searching on the internet how attention mechanism works and usually you can find a couple of really good medium or data science towards data science I think it's called like blocks and those are super useful they they they usually do a really good visual explanation and they will help you start start building that knowledge from top to bottom like strengthening that knowledge pyramid you have and so after after reading a couple of papers of a couple of blogs I found this this amazing super amazing blog by this guy Jay Elomar and it helped me a lot understand how the thing works and that's that's something really interesting so even the best researchers in the field are are sometimes less useful for you as somebody who's just starting to learn something it's they are less useful than just some guy that did a really nice visualization on the internet and so don't get stuck with the like celebrities and try and find a good resource for your particular level so not everything even though something is world content like world-class content that doesn't mean it will be the best thing for you at the moment at your current skill level so anyways this this blog does a really awesome job at explaining how attention mechanism works and it helped me a lot understand how the thing works on the high level and then the next step was to read the papers so I read the original paper by Yoshua Bengio and his collaborator and it's called neural machine translation by jointly learning to align and translate so it's a it's a mouthful my my god so after that I found I somehow found I don't remember every single detail but I found this paper effective approaches to attention-based neural machine translation and that paper introduced the concept of a local attention mechanism and a couple of other things but basically and also it introduced new ideas around the scoring functions how you can use some different scoring functions so not only the concat which was used in the original paper but also the dot product one the general attention the general scoring function etc so those are just some maybe boring details which I don't want to get into right now but I'm just trying to kind of show you how my how my process works after I felt I understand attention really thoroughly I knew I had to understand the self attention concept which was being which was the concrete thing I was being used in transformers so again I started with blocks high level resources first then going into depth so again found some really nice blocks some not so useful ones and but again J LMR had awesome awesome block which helped me understand how the self self attention mechanism works at least on the high level but he actually also goes into depth so it's a I mean I think the the series of blocks he created around transformers and NLP are probably the best resources I've seen in my life so you do have to check out this guy he's really awesome kudos to him and I also watched a couple of videos to just to get like because it was my first time I'm doing this I'm as I said I'm not so much internal P I am now a lot more but I wasn't two weeks ago so I did some I watched some videos like computer file and a couple of other channels code Emporium has a really nice visualization kudos to him also and that helped me get a nice understanding of the self attention mechanism and after after doing the the high level stuff again I just read the transformer paper again and now it was a lot more it was much more clear for me how stuff works I understood the details also and I just the high level picture so that was really nice yeah again I searched for some more resources and found some blocks that have a really nice recap of various attention mechanisms so that's that's the way I work so I start with high level go into depth and along the way I do find a bunch of useful resources and I just try and organize all of those into into my one note now after reading all of the this and having a thorough understanding of transformer I wanted to you know I wanted to figure out is there something else aside from BERT and GPT family of transformers so you probably heard of those two they're super famous and I just wanted to get an understanding what's of what's out there what's the current state of the art so what I did I'll just skip over this so what I did is I started searching for some overview blocks of the NLP and the currently best models so that's what I did next and I figure out basically only have BERT or BERT derivatives like Alberta Roberta and like but a bunch of other models and you have the GPT family GPT 1 2 & 3 by open AI which are crashing it basically at this moment so that's how I organize my my next things I'm gonna learn so I started learning about BERT I started learning about GPTs then again reading and learning about BERT I figure out I don't have some prerequisite knowledge and I mean that's how learning basically works you it's it's super nonlinear I'm trying to to just lead you through my like process but hopefully will be useful for you also and I figure out I don't know anything about word embeddings and I mean I did have some high-level knowledge like the nice semantic properties where you can basically do some arithmetic on the embeddings so it's a similar thing that you can find in the GAN literature with latent vectors there so doing some arithmetic in the latent space in the like in GANs you can create some you can get some interesting results in the image space and again I understood I have to understand these word embeddings a bit better so I started investigating what word to back is what glovey I think it's pronounced like glovey I don't know is and again found some again J LMR had awesome blog so it helped me a lot I read like the original research papers in 2013 and I also read the original neural language model by Ben Geo that was written in 2003 which was the I think the first model where they employed a neural network to to solve the language modeling task previously they used Engrams and some other techniques from statistical machine learning to to kind of crack this problem of language modeling where basically the task is if you don't know is to predict what the next word is given the previous sentence or previous couple of words so that's that's basically one of the dependencies I had bird embeddings and then again this there is this thing called the tokenization because you usually don't do anymore you don't do embedding for every single word you first tokenize the whole sentence and there is a whole science around tokenization and the thing is Bert used word piece so that's why I had this dependency and I wanted to learn what took a bit more about tokenization and turn out it's a complete like a separate research field and it's super under underestimated but it's used everywhere and you basically have character level embeddings you have sub word embeddings and you have word embeddings and most of the current state-of-the-art models like GPT family of models and Bert they use sub word embeddings and such as like the BPE algorithm that's the byte pairing I think byte pairing coding and also you have things like word piece things like sentence piece and I'm just throwing random names at you right now but you get the point like tokenization is really important and it's being used everywhere so those were most of the dependencies I had on Bert and again I searched for some blogs I found some official Google blogs I also found J.L.Lemmer's blog on Bert again super useful as I said Code Emporium had nice video and finally I reading the the J.L.Lemmer's blog I saw a couple of more things I have to understand before I dwell into Bert and those were specifically contextualized word embeddings so the thing with word embeddings is they're super shallow representations so that's something if you're from the computer vision field that's something like having only the the edge feature detector instead of the full full rich representation which the CNNs provide you with which start with maybe some simpler shapes and edges and stuff and they get to more abstract and more semantically meaningful representations at the deeper layers of the network so the problem with word embeddings is the following they encode the same exact embedding vector for a word regardless of the context of the surrounding words and that's just not how the things were in the human language so the thing is you say for example if you have a word Queen it depends on the context whether that the Queen is maybe a Queen bee or like Queen like a royalty and the context is basically determined by the surrounding words so you want to have a different ebony vector for the word Queen depending on the context whereas the word to back and glovey methods they they basically always encoded the word with the same vector and that's something that we now know is not the best thing to do and we have so there were methods that followed the word to back and glovey and that's this Kovie that the it's contextualized vectors and then you have Elmo embeddings from language models which finally starting including the like the context of the surrounding words and that was super important for for what followed along and that's and that's bird and GPT's GPT models one crucial thing that happened aside from from from these contextualized word representations is this ULM fit universal language model where they showed that you can do fine-tuning in NLP as well so previously we all knew the computer vision we had the computer vision had this image net moment in 2012 where basically we figure out we can use a pre-trained models on the image net data set to to later just fine-tune and do many of other downstream tasks like maybe image classification on some other data sets where you don't have as much data as you have in the image image net so that's the so we finally have the same ability to do the same thing in NLP and that's to do fine-tuning and just pre-train your language model on like a huge corpus of text and then just fine-tune it for a different downstream NLP tasks and ULM fit was one of the papers they kind of promoted that idea and started this trend in NLP finally I read the the BERT paper and recently a couple of days ago I started reading about GPT again JLMR has some really nice blogs and and there are some nice blogs official blogs from open AI as well and papers or something that's when you when you finish up with those you can you can go and study some some papers as always so after I finished the two important topics and that's BERT and GPT my idea is like starting this Monday to to implement the transformer the original transformer paper from scratch because as it turned out basically both BERT and GPT basically took one part of the original transformer model so BERT took the encoder part GPTs took the decoder part of the original transformer and they just kind of increased it and training some bigger data and yeah that's that's pretty much it so I'm gonna implement it and then I have this section later work where like reading through those blogs blogs and overview blogs and bunch of different resources I already showed you I kind of accumulated some things I know it'd be useful to to read them to go through them and one of those is like Alberta like some BERT derivative and then that there is this Excel net and Google t5 and a bunch of models so anyways that's how I work I do some theory then I do some coding I take some time to let that theory kind of sink down and let it go through my fingers by coding the things from scratch and then again I shift my attention to theory I do some more theory and maybe I'll do some another project after I finish with the second second iteration of this theory exploration whatever let me let me finish up with this so I always kind of try and have a bigger picture of how this all fits into the next things I'm going to do and the next area I wanna I wanna research a bit more is graph neural networks GNNs for short and so I just have some check boxes some things I have to to understand before I proceed and start digging into that new topic so I basically already learned all of these yeah let me check I don't still don't have a clear picture of what blue metric is I'm gonna read read about it a bit later so anyways hopefully you found this video useful it's the first time I created a video like this it's a bit long format and it's not as curated so hopefully you did learn something useful here if you did go ahead and subscribe to my channel hit that bell icon so that you get notified when I upload a new video and let me know if you have any questions in the comments I'd like to answer all of those so anyways have a nice day keep learning
[{"start": 0.0, "end": 5.28, "text": " What's up folks? So in this video I had this idea of sharing with you how I"}, {"start": 5.28, "end": 10.44, "text": " approach learning a new deep learning topic. So over the past seven months I've been"}, {"start": 10.44, "end": 15.040000000000001, "text": " learning about neural style transfer for maybe three, four months. That took a"}, {"start": 15.040000000000001, "end": 21.6, "text": " while because I really wanted to get a familiarity with PyTorch and yeah,"}, {"start": 21.6, "end": 27.12, "text": " basically PyTorch. Then I did some Deep Dream project and that took maybe a"}, {"start": 27.12, "end": 33.88, "text": " month and a half and my last big deep learning topic exploration was"}, {"start": 33.88, "end": 39.36, "text": " Verganz and I did those for maybe two months. Open sourced a project there and"}, {"start": 39.36, "end": 45.28, "text": " I'm currently doing Transformers and I'll be walking you through my OneNote and"}, {"start": 45.28, "end": 49.28, "text": " how I accumulate my notes and how I approach organizing information and"}, {"start": 49.28, "end": 55.24, "text": " trying to get better at whatever topic I choose to study at the present."}, {"start": 55.24, "end": 60.400000000000006, "text": " So as I said I basically started with neural style transfer earlier this year"}, {"start": 60.400000000000006, "end": 66.28, "text": " and what I did was I first started reading lots of research papers and I do"}, {"start": 66.28, "end": 69.6, "text": " understand that if you're a beginner you won't start with research papers but"}, {"start": 69.6, "end": 73.3, "text": " there are lots of blogs and some high-level resources which you could"}, {"start": 73.3, "end": 78.16, "text": " leverage to get a better understanding of the field and so I can show you here"}, {"start": 78.16, "end": 87.12, "text": " I basically yeah I just started with the original neural style transfer paper"}, {"start": 87.12, "end": 93.96, "text": " and that's this one by Gatiss and so basically while reading the paper you"}, {"start": 93.96, "end": 99.28, "text": " you find some references and reading blogs you figure out there are some"}, {"start": 99.28, "end": 104.16, "text": " better methods now and then and that's how you start slowly accumulating good"}, {"start": 104.16, "end": 109.64, "text": " research papers to read and also like using simply using Google and figuring"}, {"start": 109.64, "end": 115.52, "text": " out how many citations a paper has that's also one of the probably good"}, {"start": 115.52, "end": 120.08, "text": " indicators of whether you should be reading it or not. So as you can see here"}, {"start": 120.08, "end": 124.75999999999999, "text": " so I've been reading like I read maybe 18 papers on neural style transfer and"}, {"start": 124.75999999999999, "end": 130.56, "text": " also I don't I don't read every single paper thoroughly some of those I just"}, {"start": 130.56, "end": 134.08, "text": " kind of glimpse and so I organized it like this so I ran through those"}, {"start": 134.08, "end": 138.72, "text": " whatever like silly name just a directory where I dump the papers which"}, {"start": 138.72, "end": 145.04, "text": " I just kind of have some overview but I am not thoroughly familiar with those"}, {"start": 145.04, "end": 152.28, "text": " and these are kind of some papers which are on my backlog and I don't think I'll"}, {"start": 152.28, "end": 158.48000000000002, "text": " be I'll have time to to go through these but yeah they are they are still"}, {"start": 158.48, "end": 164.51999999999998, "text": " here anyways so after running through the theory what I did was basically"}, {"start": 164.51999999999998, "end": 171.32, "text": " started implementing stuff from scratch and that's super important so theory"}, {"start": 171.32, "end": 176.39999999999998, "text": " must be complemented with code otherwise it will be it won't be the same you you"}, {"start": 176.39999999999998, "end": 180.01999999999998, "text": " won't have a deep understanding of how stuff works because devil is in the"}, {"start": 180.01999999999998, "end": 184.1, "text": " details like the first paper I started implementing was Gadeus's original"}, {"start": 184.1, "end": 188.28, "text": " paper and I thought I knew everything works but I didn't turned out I didn't"}, {"start": 188.28, "end": 194.76, "text": " and so I struggle with some details like should you use 0 to 25 to 55 range or 0"}, {"start": 194.76, "end": 200.4, "text": " to 1 range and there's so many of those small details which you don't think"}, {"start": 200.4, "end": 205.6, "text": " about when you're just reading theory so what I did was I implemented the"}, {"start": 205.6, "end": 211.52, "text": " original paper here then I I implemented some Johnson's paper which didn't use"}, {"start": 211.52, "end": 215.88, "text": " the optimization method they just use convolutional neural network to you"}, {"start": 215.88, "end": 219.35999999999999, "text": " basically input an image it passes through the convolutional neural net and"}, {"start": 219.35999999999999, "end": 226.12, "text": " out goes the stylized version of that same image and finally I did play with"}, {"start": 226.12, "end": 231.84, "text": " videos so I did implement like a naive approach of neural style transfer for"}, {"start": 231.84, "end": 236.56, "text": " videos naive in the sense that I don't have any temporal constraints included"}, {"start": 236.56, "end": 241.28, "text": " so it's kind of jittery when you go from frame to frame but I did play with some"}, {"start": 241.28, "end": 246.76, "text": " like models which actually during the training include the temporal loss and"}, {"start": 246.76, "end": 251.72, "text": " so the result is much more stable once you start doing the inference let's go"}, {"start": 251.72, "end": 254.88, "text": " and take a look at a couple of the projects implemented so the first one"}, {"start": 254.88, "end": 259.88, "text": " was so the optimization method by Gadeus and you'll look the original neural"}, {"start": 259.88, "end": 264.48, "text": " style transfer algorithm and as you can see here I do think I'm a big believer"}, {"start": 264.48, "end": 268.9, "text": " that you should create a really good readme file otherwise people will get"}, {"start": 268.9, "end": 273.47999999999996, "text": " confused nobody wants to have to go deep inside the code you know to understand"}, {"start": 273.47999999999996, "end": 278.32, "text": " what's happening you should present what your what your code is doing so that"}, {"start": 278.32, "end": 282.28, "text": " it's easy for the others to to know straight ahead whether they want to play"}, {"start": 282.28, "end": 285.76, "text": " with your code or not and you should put some basic instructions how to set up"}, {"start": 285.76, "end": 292.03999999999996, "text": " your code how to use your code and maybe how to debug and experiment and whatever"}, {"start": 292.03999999999996, "end": 296.88, "text": " additional things you have and also like putting acknowledgments is really nice"}, {"start": 296.88, "end": 300.96, "text": " because you always build up on somebody else's work or you use it as an"}, {"start": 300.96, "end": 306.56, "text": " inspiration to build your own work and many do not admit this but I think it's"}, {"start": 306.56, "end": 311.44, "text": " like everybody knows that everybody's using other people's code and either"}, {"start": 311.44, "end": 316.38, "text": " building off of it or using it as an inspiration so that was the that was the"}, {"start": 316.38, "end": 320.64, "text": " first project I did and then the second one as I mentioned was Johnson's same"}, {"start": 320.64, "end": 326.76, "text": " similar thing it's this one is much faster it can run almost real-time but"}, {"start": 326.76, "end": 335.59999999999997, "text": " it's has a bit worse quality than than the Gattas project and finally this is"}, {"start": 335.59999999999997, "end": 340.15999999999997, "text": " the video project I was talking about so what I did here additionally was I"}, {"start": 340.15999999999997, "end": 345.64, "text": " included a small part that will actually segment the the the user that's in the"}, {"start": 345.64, "end": 350.88, "text": " frame so so so it's basically built for a specific setup and that's a person"}, {"start": 350.88, "end": 354.56, "text": " talking in front of the camera like I'm doing right now and it does a pretty"}, {"start": 354.56, "end": 359.68, "text": " decent job in that setup as you can see here so what you can do after you find"}, {"start": 359.68, "end": 363.15999999999997, "text": " the segmentation mask is you can apply different styles to either background or"}, {"start": 363.15999999999997, "end": 370.24, "text": " foreground and that's that's a useful useful nice thing to have and here is a"}, {"start": 370.24, "end": 376.76, "text": " small GIF I created using that project so you can go ahead and play with these"}, {"start": 376.76, "end": 381.56, "text": " so the main idea here is just to show you how how I personally work and maybe"}, {"start": 381.56, "end": 385.04, "text": " you'll find some inspiration hopefully do if you do please leave a comment"}, {"start": 385.04, "end": 390.68, "text": " whether this video helped you or did not next thing I did as I mentioned was"}, {"start": 390.68, "end": 395.96000000000004, "text": " deep dream and now you may wonder why is this guy doing this nice stylistic"}, {"start": 395.96, "end": 400.32, "text": " projects I mean you don't learn anything pragmatic right so the thing is in"}, {"start": 400.32, "end": 404.79999999999995, "text": " Microsoft I'm doing lots of really practical stuff and so I have this"}, {"start": 404.79999999999995, "end": 409.0, "text": " luxury of just playing around with the different techniques different research"}, {"start": 409.0, "end": 414.59999999999997, "text": " areas in my free time so the thing is you learn a lot doing these projects so"}, {"start": 414.59999999999997, "end": 418.64, "text": " for example doing neural style transfer you you'll learn about the concept called"}, {"start": 418.64, "end": 423.79999999999995, "text": " perceptual loss for example that's used all around the place in other areas also"}, {"start": 423.8, "end": 427.48, "text": " for example the Johnson the paper I mentioned with the like the Johnson's"}, {"start": 427.48, "end": 432.96000000000004, "text": " paper he the architecture he used for the generation of neural style transfer"}, {"start": 432.96000000000004, "end": 438.40000000000003, "text": " images is something that was was being really used in and I've seen it reused in"}, {"start": 438.40000000000003, "end": 445.16, "text": " bunch of other papers so you do get to extract some useful information from one"}, {"start": 445.16, "end": 450.64, "text": " field and transfer it to another so one additionally so there's this thing"}, {"start": 450.64, "end": 454.96, "text": " called adaptive instant normalization and it was invented in the context of"}, {"start": 454.96, "end": 459.56, "text": " the neural style transfer field but I recently saw it while researching GANs"}, {"start": 459.56, "end": 464.08, "text": " and the famous style again architecture is actually using adaptive instance"}, {"start": 464.08, "end": 468.71999999999997, "text": " normalization so that's that's the cross-semination I was talking about so"}, {"start": 468.71999999999997, "end": 474.56, "text": " basically do learn a lot by doing some maybe unrelated field but it is"}, {"start": 474.56, "end": 479.88, "text": " related to whatever you'll be doing in the future if you if you decide to keep"}, {"start": 479.88, "end": 486.0, "text": " doing deep learning anyways this is how deep dream algorithm looks like I think"}, {"start": 486.0, "end": 490.84, "text": " it's got really decent images just super exciting to see that the neural"}, {"start": 490.84, "end": 496.38, "text": " network itself has this kind of power inside and that you can achieve this by"}, {"start": 496.38, "end": 501.96, "text": " just using simple optimization methods and CNNs and I did create some gifts"}, {"start": 501.96, "end": 509.76, "text": " here and that's all awesome jumping into GANs quickly so that was my final deep"}, {"start": 509.76, "end": 516.0, "text": " learning topic I researched and again read bunch of papers that's usually how"}, {"start": 516.0, "end": 520.4399999999999, "text": " I start exploring the field actually I'm lying I actually start usually start"}, {"start": 520.4399999999999, "end": 523.88, "text": " with high-level resources but I'll get to details a bit later with transformers"}, {"start": 523.88, "end": 530.2, "text": " so after reading papers I just developed from scratch like the vanilla the"}, {"start": 530.2, "end": 534.28, "text": " original generative adversarial network then I did some conditional GANs where"}, {"start": 534.28, "end": 538.88, "text": " you can use you can condition which class you want to generate so in"}, {"start": 538.88, "end": 545.12, "text": " particular I did amnest and I used amnest data here so what I achieved was"}, {"start": 545.12, "end": 549.64, "text": " that I can basically choose which class I want to generate so here you can see I"}, {"start": 549.64, "end": 556.84, "text": " can condition the generator to create only zeros or only ones etc and finally"}, {"start": 556.84, "end": 563.72, "text": " I developed this DC GAN paper which was super popular and it's super famous"}, {"start": 563.72, "end": 570.48, "text": " because it basically created this Cambrian explosion of new research"}, {"start": 570.48, "end": 577.48, "text": " papers that were using DC GAN as a building block and it really accelerated"}, {"start": 577.48, "end": 583.88, "text": " the field as such so like doing doing this I learned a lot and that's"}, {"start": 583.88, "end": 587.9200000000001, "text": " something I recommend to everybody that's that's a really good way to learn"}, {"start": 587.9200000000001, "end": 593.6, "text": " start with theory don't dwell too much on theory maybe two weeks like tops and"}, {"start": 593.6, "end": 599.0400000000001, "text": " then start start coding it up from scratch I think I mentioned this I still"}, {"start": 599.0400000000001, "end": 602.36, "text": " didn't so I'm currently doing transformers I'm in this phase of"}, {"start": 602.36, "end": 607.26, "text": " exploring the theory that's behind transformers so I still don't have as"}, {"start": 607.26, "end": 611.84, "text": " you can see here I don't have the project implemented yet but I'll start"}, {"start": 611.84, "end": 616.44, "text": " doing it like this Monday I think so hopefully you folks will be able to play"}, {"start": 616.44, "end": 621.12, "text": " with the code once I finish it up in maybe two weeks time so let me let me"}, {"start": 621.12, "end": 626.24, "text": " walk you through my one note and show you how I approach learning a new"}, {"start": 626.24, "end": 630.68, "text": " deep learning fields I don't start with research papers usually unless I'm"}, {"start": 630.68, "end": 634.12, "text": " really familiar with the field I do start with some high-level resources and"}, {"start": 634.12, "end": 639.84, "text": " let's just check out how it looks like so if I open my one note you can see a"}, {"start": 639.84, "end": 646.0, "text": " bunch of like resources I've accumulated over the last two weeks actually because"}, {"start": 646.0, "end": 651.24, "text": " I've been doing this pretty much every single day and yeah let me start"}, {"start": 651.24, "end": 656.24, "text": " slowly digesting this for you so first a little bit of context how I started and"}, {"start": 656.24, "end": 660.24, "text": " why I started doing transformers so first of all you all heard of GPT-3 and"}, {"start": 660.24, "end": 665.08, "text": " the super famous language model by OpenAI and the cool stuff it's been"}, {"start": 665.08, "end": 669.4, "text": " creating over the last couple of months and lots of hype around there so that"}, {"start": 669.4, "end": 672.6, "text": " wasn't actually the reason I started doing transformers even though they were"}, {"start": 672.6, "end": 676.8000000000001, "text": " on my backlog I would I would have started learning transformers at one"}, {"start": 676.8000000000001, "end": 680.64, "text": " point of time but it's just kind of happened sooner because what happened is"}, {"start": 680.64, "end": 685.6, "text": " this visual transformer and it was published I think around 3rd of October"}, {"start": 685.6, "end": 689.96, "text": " on this machine learning conference called iClear I think that's how it's"}, {"start": 689.96, "end": 695.8000000000001, "text": " pronounced and basically what it did it showed that transformers are more"}, {"start": 695.8000000000001, "end": 701.24, "text": " powerful than CNNs in the field of computer vision in specifically"}, {"start": 701.24, "end": 706.28, "text": " in the big data regime and it kind of blew my mind off and I was like I have"}, {"start": 706.28, "end": 711.08, "text": " to start doing this sooner so there was a short background story why I started"}, {"start": 711.08, "end": 716.44, "text": " doing transformers and what I first did was I just opened the paper visual"}, {"start": 716.44, "end": 720.88, "text": " transformer and I wasn't an NLP guy I was a computer vision guy so I've been"}, {"start": 720.88, "end": 724.44, "text": " spending most of my time over the last couple of years doing computer vision and"}, {"start": 724.44, "end": 729.64, "text": " I started reading it and I basically immediately saw it has a really strong"}, {"start": 729.64, "end": 733.68, "text": " dependency on the original transformer paper so that's what I did I took a step"}, {"start": 733.68, "end": 737.56, "text": " back I opened the transformer paper and I've read it from top to bottom and"}, {"start": 737.56, "end": 742.68, "text": " basically after finishing the paper then I read the visual transformer and I"}, {"start": 742.68, "end": 746.68, "text": " still didn't have a clue of what's happening many details I mean I did have"}, {"start": 746.68, "end": 750.36, "text": " a rough understanding what's happening but I didn't have many details so I"}, {"start": 750.36, "end": 755.46, "text": " decided I have to take a step even one step back and start learning about the"}, {"start": 755.46, "end": 761.4000000000001, "text": " attention mechanism and some other dependencies that those papers had like"}, {"start": 761.4000000000001, "end": 767.2, "text": " like word embeddings and concepts like like that one let's finally start"}, {"start": 767.2, "end": 770.88, "text": " walking through the the one note so as you can see here that this section is"}, {"start": 770.88, "end": 775.12, "text": " attention as I mentioned I figure out I'll need to understand the tension"}, {"start": 775.12, "end": 779.2800000000001, "text": " mechanism thoroughly so what I'd start what I started doing was first just"}, {"start": 779.2800000000001, "end": 784.32, "text": " searching on the internet how attention mechanism works and usually you can find"}, {"start": 784.32, "end": 788.2800000000001, "text": " a couple of really good medium or data science towards data science I think"}, {"start": 788.2800000000001, "end": 793.5600000000001, "text": " it's called like blocks and those are super useful they they they usually do"}, {"start": 793.5600000000001, "end": 798.5600000000001, "text": " a really good visual explanation and they will help you start start building"}, {"start": 798.5600000000001, "end": 803.6400000000001, "text": " that knowledge from top to bottom like strengthening that knowledge pyramid you"}, {"start": 803.6400000000001, "end": 810.5600000000001, "text": " have and so after after reading a couple of papers of a couple of blogs I found"}, {"start": 810.56, "end": 818.8399999999999, "text": " this this amazing super amazing blog by this guy Jay Elomar and it helped me a"}, {"start": 818.8399999999999, "end": 823.7199999999999, "text": " lot understand how the thing works and that's that's something really"}, {"start": 823.7199999999999, "end": 828.68, "text": " interesting so even the best researchers in the field are are sometimes less"}, {"start": 828.68, "end": 833.88, "text": " useful for you as somebody who's just starting to learn something it's they"}, {"start": 833.88, "end": 837.76, "text": " are less useful than just some guy that did a really nice visualization on the"}, {"start": 837.76, "end": 842.4, "text": " internet and so don't get stuck with the like celebrities and try and find a good"}, {"start": 842.4, "end": 847.2, "text": " resource for your particular level so not everything even though something is"}, {"start": 847.2, "end": 850.6, "text": " world content like world-class content that doesn't mean it will be the best"}, {"start": 850.6, "end": 857.08, "text": " thing for you at the moment at your current skill level so anyways this this"}, {"start": 857.08, "end": 861.84, "text": " blog does a really awesome job at explaining how attention mechanism"}, {"start": 861.84, "end": 868.0, "text": " works and it helped me a lot understand how the thing works on the high level"}, {"start": 868.0, "end": 872.1, "text": " and then the next step was to read the papers so I read the original paper by"}, {"start": 872.1, "end": 877.84, "text": " Yoshua Bengio and his collaborator and it's called neural machine translation"}, {"start": 877.84, "end": 881.44, "text": " by jointly learning to align and translate so it's a it's a mouthful my"}, {"start": 881.44, "end": 889.2, "text": " my god so after that I found I somehow found I don't remember every single"}, {"start": 889.2, "end": 892.4000000000001, "text": " detail but I found this paper effective approaches to attention-based neural"}, {"start": 892.4000000000001, "end": 897.4000000000001, "text": " machine translation and that paper introduced the concept of a local"}, {"start": 897.4000000000001, "end": 903.84, "text": " attention mechanism and a couple of other things but basically and also it"}, {"start": 903.84, "end": 909.88, "text": " introduced new ideas around the scoring functions how you can use some different"}, {"start": 909.88, "end": 914.36, "text": " scoring functions so not only the concat which was used in the original paper but"}, {"start": 914.36, "end": 918.6800000000001, "text": " also the dot product one the general attention the general scoring function"}, {"start": 918.68, "end": 922.92, "text": " etc so those are just some maybe boring details which I don't want to get into"}, {"start": 922.92, "end": 928.12, "text": " right now but I'm just trying to kind of show you how my how my process works"}, {"start": 928.12, "end": 933.5999999999999, "text": " after I felt I understand attention really thoroughly I knew I had to"}, {"start": 933.5999999999999, "end": 937.0799999999999, "text": " understand the self attention concept which was being which was the concrete"}, {"start": 937.0799999999999, "end": 941.92, "text": " thing I was being used in transformers so again I started with blocks high"}, {"start": 941.92, "end": 946.76, "text": " level resources first then going into depth so again found some really nice"}, {"start": 946.76, "end": 954.12, "text": " blocks some not so useful ones and but again J LMR had awesome awesome block"}, {"start": 954.12, "end": 960.2, "text": " which helped me understand how the self self attention mechanism works at least"}, {"start": 960.2, "end": 964.84, "text": " on the high level but he actually also goes into depth so it's a I mean I think"}, {"start": 964.84, "end": 971.04, "text": " the the series of blocks he created around transformers and NLP are probably"}, {"start": 971.04, "end": 975.56, "text": " the best resources I've seen in my life so you do have to check out this guy he's"}, {"start": 975.56, "end": 982.5999999999999, "text": " really awesome kudos to him and I also watched a couple of videos to just to"}, {"start": 982.5999999999999, "end": 987.3199999999999, "text": " get like because it was my first time I'm doing this I'm as I said I'm not so"}, {"start": 987.3199999999999, "end": 994.16, "text": " much internal P I am now a lot more but I wasn't two weeks ago so I did some I"}, {"start": 994.16, "end": 998.8199999999999, "text": " watched some videos like computer file and a couple of other channels code"}, {"start": 998.8199999999999, "end": 1005.04, "text": " Emporium has a really nice visualization kudos to him also and that helped me get"}, {"start": 1005.04, "end": 1011.7199999999999, "text": " a nice understanding of the self attention mechanism and after after"}, {"start": 1011.7199999999999, "end": 1018.0799999999999, "text": " doing the the high level stuff again I just read the transformer paper again"}, {"start": 1018.0799999999999, "end": 1022.76, "text": " and now it was a lot more it was much more clear for me how stuff works I"}, {"start": 1022.76, "end": 1027.76, "text": " understood the details also and I just the high level picture so that was"}, {"start": 1027.76, "end": 1035.56, "text": " really nice yeah again I searched for some more resources and found some blocks"}, {"start": 1035.56, "end": 1040.24, "text": " that have a really nice recap of various attention mechanisms so that's that's"}, {"start": 1040.24, "end": 1046.4, "text": " the way I work so I start with high level go into depth and along the way I"}, {"start": 1046.4, "end": 1051.1, "text": " do find a bunch of useful resources and I just try and organize all of those"}, {"start": 1051.1, "end": 1056.92, "text": " into into my one note now after reading all of the this and having a thorough"}, {"start": 1056.92, "end": 1061.5600000000002, "text": " understanding of transformer I wanted to you know I wanted to figure out is there"}, {"start": 1061.5600000000002, "end": 1066.52, "text": " something else aside from BERT and GPT family of transformers so you probably"}, {"start": 1066.52, "end": 1071.2, "text": " heard of those two they're super famous and I just wanted to get an"}, {"start": 1071.2, "end": 1074.8000000000002, "text": " understanding what's of what's out there what's the current state of the art so"}, {"start": 1074.8000000000002, "end": 1081.72, "text": " what I did I'll just skip over this so what I did is I started searching for"}, {"start": 1081.72, "end": 1086.28, "text": " some overview blocks of the NLP and the currently best models so that's what I"}, {"start": 1086.28, "end": 1092.6399999999999, "text": " did next and I figure out basically only have BERT or BERT derivatives like"}, {"start": 1092.6399999999999, "end": 1097.52, "text": " Alberta Roberta and like but a bunch of other models and you have the GPT family"}, {"start": 1097.52, "end": 1102.24, "text": " GPT 1 2 & 3 by open AI which are crashing it basically at this moment so"}, {"start": 1102.24, "end": 1106.6, "text": " that's how I organize my my next things I'm gonna learn so I started learning"}, {"start": 1106.6, "end": 1112.28, "text": " about BERT I started learning about GPTs then again reading and learning about"}, {"start": 1112.28, "end": 1119.3999999999999, "text": " BERT I figure out I don't have some prerequisite knowledge and I mean that's"}, {"start": 1119.3999999999999, "end": 1124.3999999999999, "text": " how learning basically works you it's it's super nonlinear I'm trying to to"}, {"start": 1124.3999999999999, "end": 1129.76, "text": " just lead you through my like process but hopefully will be useful for you"}, {"start": 1129.76, "end": 1134.44, "text": " also and I figure out I don't know anything about word embeddings and I"}, {"start": 1134.44, "end": 1138.12, "text": " mean I did have some high-level knowledge like the nice semantic"}, {"start": 1138.12, "end": 1142.8, "text": " properties where you can basically do some arithmetic on the embeddings so"}, {"start": 1142.8, "end": 1146.76, "text": " it's a similar thing that you can find in the GAN literature with latent"}, {"start": 1146.76, "end": 1152.6799999999998, "text": " vectors there so doing some arithmetic in the latent space in the like in GANs"}, {"start": 1152.6799999999998, "end": 1158.28, "text": " you can create some you can get some interesting results in the image space"}, {"start": 1158.28, "end": 1166.1999999999998, "text": " and again I understood I have to understand these word embeddings a bit"}, {"start": 1166.2, "end": 1173.78, "text": " better so I started investigating what word to back is what glovey I think it's"}, {"start": 1173.78, "end": 1180.8, "text": " pronounced like glovey I don't know is and again found some again J LMR had"}, {"start": 1180.8, "end": 1186.68, "text": " awesome blog so it helped me a lot I read like the original research papers"}, {"start": 1186.68, "end": 1194.28, "text": " in 2013 and I also read the original neural language model by Ben Geo that"}, {"start": 1194.28, "end": 1199.8799999999999, "text": " was written in 2003 which was the I think the first model where they"}, {"start": 1199.8799999999999, "end": 1204.76, "text": " employed a neural network to to solve the language modeling task previously"}, {"start": 1204.76, "end": 1209.06, "text": " they used Engrams and some other techniques from statistical machine"}, {"start": 1209.06, "end": 1213.8, "text": " learning to to kind of crack this problem of language modeling where"}, {"start": 1213.8, "end": 1217.72, "text": " basically the task is if you don't know is to predict what the next word is"}, {"start": 1217.72, "end": 1226.3600000000001, "text": " given the previous sentence or previous couple of words so that's that's"}, {"start": 1226.3600000000001, "end": 1232.88, "text": " basically one of the dependencies I had bird embeddings and then again this"}, {"start": 1232.88, "end": 1237.92, "text": " there is this thing called the tokenization because you usually don't"}, {"start": 1237.92, "end": 1244.28, "text": " do anymore you don't do embedding for every single word you first tokenize the"}, {"start": 1244.28, "end": 1248.72, "text": " whole sentence and there is a whole science around tokenization and the"}, {"start": 1248.72, "end": 1253.72, "text": " thing is Bert used word piece so that's why I had this dependency and I wanted"}, {"start": 1253.72, "end": 1256.92, "text": " to learn what took a bit more about tokenization and turn out it's a"}, {"start": 1256.92, "end": 1262.76, "text": " complete like a separate research field and it's super under underestimated but"}, {"start": 1262.76, "end": 1268.3999999999999, "text": " it's used everywhere and you basically have character level embeddings you"}, {"start": 1268.3999999999999, "end": 1272.04, "text": " have sub word embeddings and you have word embeddings and most of the current"}, {"start": 1272.04, "end": 1276.6, "text": " state-of-the-art models like GPT family of models and Bert they use sub word"}, {"start": 1276.6, "end": 1282.28, "text": " embeddings and such as like the BPE algorithm that's the byte pairing I"}, {"start": 1282.28, "end": 1288.96, "text": " think byte pairing coding and also you have things like word piece things like"}, {"start": 1288.96, "end": 1292.52, "text": " sentence piece and I'm just throwing random names at you right now but you"}, {"start": 1292.52, "end": 1297.08, "text": " get the point like tokenization is really important and it's being used"}, {"start": 1297.08, "end": 1305.24, "text": " everywhere so those were most of the dependencies I had on Bert and again I"}, {"start": 1305.24, "end": 1309.9199999999998, "text": " searched for some blogs I found some official Google blogs I also found"}, {"start": 1309.9199999999998, "end": 1315.6799999999998, "text": " J.L.Lemmer's blog on Bert again super useful as I said Code Emporium had nice"}, {"start": 1315.6799999999998, "end": 1323.28, "text": " video and finally I reading the the J.L.Lemmer's blog I saw a couple of more"}, {"start": 1323.28, "end": 1327.24, "text": " things I have to understand before I dwell into Bert and those were"}, {"start": 1327.24, "end": 1331.96, "text": " specifically contextualized word embeddings so the thing with word"}, {"start": 1331.96, "end": 1335.8799999999999, "text": " embeddings is they're super shallow representations so that's something if"}, {"start": 1335.8799999999999, "end": 1340.04, "text": " you're from the computer vision field that's something like having only the"}, {"start": 1340.04, "end": 1346.62, "text": " the edge feature detector instead of the full full rich representation which the"}, {"start": 1346.62, "end": 1351.28, "text": " CNNs provide you with which start with maybe some simpler shapes and edges and"}, {"start": 1351.28, "end": 1355.76, "text": " stuff and they get to more abstract and more semantically meaningful"}, {"start": 1355.76, "end": 1360.2, "text": " representations at the deeper layers of the network so the problem with word"}, {"start": 1360.2, "end": 1365.28, "text": " embeddings is the following they encode the same exact embedding vector for a"}, {"start": 1365.28, "end": 1370.12, "text": " word regardless of the context of the surrounding words and that's just not"}, {"start": 1370.12, "end": 1374.36, "text": " how the things were in the human language so the thing is you say for"}, {"start": 1374.36, "end": 1380.44, "text": " example if you have a word Queen it depends on the context whether that the"}, {"start": 1380.44, "end": 1385.2, "text": " Queen is maybe a Queen bee or like Queen like a royalty and the context is"}, {"start": 1385.2, "end": 1389.4, "text": " basically determined by the surrounding words so you want to have a different"}, {"start": 1389.4, "end": 1393.76, "text": " ebony vector for the word Queen depending on the context whereas the"}, {"start": 1393.76, "end": 1399.04, "text": " word to back and glovey methods they they basically always encoded the word"}, {"start": 1399.04, "end": 1403.5, "text": " with the same vector and that's something that we now know is not the"}, {"start": 1403.5, "end": 1408.88, "text": " best thing to do and we have so there were methods that followed the word to"}, {"start": 1408.88, "end": 1414.1200000000001, "text": " back and glovey and that's this Kovie that the it's contextualized vectors and"}, {"start": 1414.1200000000001, "end": 1419.0, "text": " then you have Elmo embeddings from language models which finally starting"}, {"start": 1419.0, "end": 1423.5200000000002, "text": " including the like the context of the surrounding words and that was super"}, {"start": 1423.5200000000002, "end": 1430.1200000000001, "text": " important for for what followed along and that's and that's bird and GPT's GPT"}, {"start": 1430.1200000000001, "end": 1436.0800000000002, "text": " models one crucial thing that happened aside from from from these"}, {"start": 1436.08, "end": 1443.6799999999998, "text": " contextualized word representations is this ULM fit universal language model"}, {"start": 1443.6799999999998, "end": 1450.84, "text": " where they showed that you can do fine-tuning in NLP as well so previously"}, {"start": 1450.84, "end": 1454.6399999999999, "text": " we all knew the computer vision we had the computer vision had this image net"}, {"start": 1454.6399999999999, "end": 1461.48, "text": " moment in 2012 where basically we figure out we can use a pre-trained models on"}, {"start": 1461.48, "end": 1467.96, "text": " the image net data set to to later just fine-tune and do many of other"}, {"start": 1467.96, "end": 1472.68, "text": " downstream tasks like maybe image classification on some other data sets"}, {"start": 1472.68, "end": 1477.2, "text": " where you don't have as much data as you have in the image image net so that's"}, {"start": 1477.2, "end": 1484.56, "text": " the so we finally have the same ability to do the same thing in NLP and that's"}, {"start": 1484.56, "end": 1489.38, "text": " to do fine-tuning and just pre-train your language model on like a huge"}, {"start": 1489.38, "end": 1494.42, "text": " corpus of text and then just fine-tune it for a different downstream NLP tasks"}, {"start": 1494.42, "end": 1500.3200000000002, "text": " and ULM fit was one of the papers they kind of promoted that idea and started"}, {"start": 1500.3200000000002, "end": 1507.7600000000002, "text": " this trend in NLP finally I read the the BERT paper and recently a couple of"}, {"start": 1507.7600000000002, "end": 1512.5200000000002, "text": " days ago I started reading about GPT again JLMR has some really nice blogs"}, {"start": 1512.52, "end": 1520.16, "text": " and and there are some nice blogs official blogs from open AI as well and"}, {"start": 1520.16, "end": 1524.76, "text": " papers or something that's when you when you finish up with those you can you can"}, {"start": 1524.76, "end": 1528.72, "text": " go and study some some papers as always so after I finished the two important"}, {"start": 1528.72, "end": 1533.92, "text": " topics and that's BERT and GPT my idea is like starting this Monday to to"}, {"start": 1533.92, "end": 1538.44, "text": " implement the transformer the original transformer paper from scratch because"}, {"start": 1538.44, "end": 1543.8, "text": " as it turned out basically both BERT and GPT basically took one part of the"}, {"start": 1543.8, "end": 1549.52, "text": " original transformer model so BERT took the encoder part GPTs took the decoder"}, {"start": 1549.52, "end": 1553.92, "text": " part of the original transformer and they just kind of increased it and"}, {"start": 1553.92, "end": 1558.48, "text": " training some bigger data and yeah that's that's pretty much it so I'm"}, {"start": 1558.48, "end": 1564.18, "text": " gonna implement it and then I have this section later work where like reading"}, {"start": 1564.18, "end": 1569.0, "text": " through those blogs blogs and overview blogs and bunch of different resources I"}, {"start": 1569.0, "end": 1574.1200000000001, "text": " already showed you I kind of accumulated some things I know it'd be useful to to"}, {"start": 1574.1200000000001, "end": 1578.72, "text": " read them to go through them and one of those is like Alberta like some BERT"}, {"start": 1578.72, "end": 1583.96, "text": " derivative and then that there is this Excel net and Google t5 and a bunch of"}, {"start": 1583.96, "end": 1593.28, "text": " models so anyways that's how I work I do some theory then I do some coding I take"}, {"start": 1593.28, "end": 1598.6399999999999, "text": " some time to let that theory kind of sink down and let it go through my"}, {"start": 1598.6399999999999, "end": 1603.8, "text": " fingers by coding the things from scratch and then again I shift my"}, {"start": 1603.8, "end": 1609.32, "text": " attention to theory I do some more theory and maybe I'll do some another"}, {"start": 1609.32, "end": 1615.6, "text": " project after I finish with the second second iteration of this theory"}, {"start": 1615.6, "end": 1621.96, "text": " exploration whatever let me let me finish up with this so I always kind of"}, {"start": 1621.96, "end": 1627.04, "text": " try and have a bigger picture of how this all fits into the next things I'm"}, {"start": 1627.04, "end": 1633.48, "text": " going to do and the next area I wanna I wanna research a bit more is graph"}, {"start": 1633.48, "end": 1639.28, "text": " neural networks GNNs for short and so I just have some check boxes some things I"}, {"start": 1639.28, "end": 1646.28, "text": " have to to understand before I proceed and start digging into that new topic so"}, {"start": 1646.28, "end": 1652.0, "text": " I basically already learned all of these yeah let me check I don't still don't"}, {"start": 1652.0, "end": 1657.08, "text": " have a clear picture of what blue metric is I'm gonna read read about it a bit"}, {"start": 1657.08, "end": 1664.76, "text": " later so anyways hopefully you found this video useful it's the first time I"}, {"start": 1664.76, "end": 1671.0, "text": " created a video like this it's a bit long format and it's not as curated so"}, {"start": 1671.0, "end": 1675.56, "text": " hopefully you did learn something useful here if you did go ahead and"}, {"start": 1675.56, "end": 1680.72, "text": " subscribe to my channel hit that bell icon so that you get notified when I"}, {"start": 1680.72, "end": 1686.08, "text": " upload a new video and let me know if you have any questions in the comments"}, {"start": 1686.08, "end": 1705.76, "text": " I'd like to answer all of those so anyways have a nice day keep learning"}]