Developing a deep learning project (case study on transformer)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I talk about how I work on my deep learning projects on the example of the transformer which I've recently open-sourced.

You'll learn about:
✔️ Snippets of my personal story
✔️ How I think about approaching a new project
✔️ Problems I encountered developing the transformer

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ My transformer implementation: https://github.com/gordicaleksa/pytorch-original-transformer
✅ The Annotated Transformer blog: http://nlp.seas.harvard.edu/2018/04/03/attention.html
✅ Original paper: https://arxiv.org/abs/1706.03762
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

0:00 Open-sourcing original transformer and why
3:00 Creating great projects takes time
5:40 My time management
9:00 My story and embracing failures
12:05 Overview of the transformer project
14:03 Data and task definition
15:57 Training loop
20:10 Problems I encountered
21:44 Beam search fun
23:10 BucketIterator fun
25:35 Optimizing things to speed up the loop
26:25 Translating from English to German and vice versa
30:00 Hardware requirements
30:47 Wrapping things up

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#transformer #attention #deeplearning
 Finally, finally I've open sourced my implementation of the original transformer paper, Attention is All You Need, and it took me like three weeks of non-stop working on this and I'm super happy. I learned a lot and hopefully you'll find it as a valuable resource as well. So yeah, let me just walk you through this, read me. This is the architecture. I have some nice visualizations here and I have two models which I've pre-trained, which I've linked. So yeah, before I get into this, like the code itself, I just want to tell you about a couple of things, like the whole journey I had learning about this. And so those of you who've been following me over the last like a couple of weeks, you know that I've been... so I've been working on transformers for more than a month now and I first started by like just reading about theory, reading like research papers for more than like two plus weeks. And then after I finished that one, I basically started coding the implementation from scratch and that took me around three weeks more or less. So let me start with why I did it. So the thing is I looked for like good resources to understand transformers but not on the like level of theory but like to be able to actually understand the code itself. And I basically only found two decent resources, good resources. Basically, I'm not counting in Hugging Face because they do have a really awesome library of transformers but they are like more for like as a black box experience type of usage and not for going through the code and understanding how stuff works. So the two resources I mentioned are the annotated transformer is the first one and then PyTorch official implementation. Now the thing with PyTorch official implementation is that like their multi-headed attention module is awesome, like functionality wise it works. But the thing is it's so generic because they got to cover a bunch of use cases and so it's so complicated to actually go through the code and understand how it works. So my idea was to just focus on this specific use case and like of this transformer model from 2017 and that's it. On the other hand, the annotated transformer block is basically suited only for researchers and also had a couple of bugs. It's a it's an overall really nice resource and helped me a lot but it's really only for researchers and they also like kind of suppose you understand the source code of Torch text which is PyTorch's like module for manipulating NLP tasks which is usually not the not the case right. So I want to I want to make a couple of points before we go and take an overview of the code and I explain you how I struggled, which problems I had, how I approached it. I won't get into every single detail because the code base is now already pretty big. I'll show you some details but before that I want to tell you that making good projects takes time. So I had I had a couple of comments. I had Rahul commenting, thanks for telling us that understanding and coding a new AI algorithm takes a few months. I always felt like people are going too fast and doing things in just days and I cannot keep up. But thank you, it gives me motivation. So thank you Rahul. The thing is I mean it's funny like my first so my first attempt at trying to reconstruct a research paper took three months. So I was coding three months non-stop and the reason being the authors did not open source their like implementation and you couldn't find the code wasn't as popular as the transformer so I couldn't find snippets of code anywhere. So what I what I ended up doing is just like trying and reconstructing the the the paper by just reading text which is super hard and it takes time and I had lots of back and forth with the authors to understand certain big details and it's like so the point I want to make here is don't fall for that learn topic huge topic X in five days. I mean that doesn't work like that and I'm just wondering I think I know why that's the case like you have you have Siraj Rawal who had videos like learn tensorflow in five minutes and I mean if you I want to tell you something I mean if you if you believe that crap you're in a really bad position because that's going to set you up for failure and why because you're basically it's basically setting up setting you up for impatience whereas you really truly need patience in order to create anything great like every single project like the projects I'm doing at Microsoft they take that's like a multi like multi like team effort that lasts at least six months one year two years three years so anything significant takes time in artificial intelligence and computer science whatever the topic it is like I have skipped creating a video last week last weekend because I was so in a flow of cutting this transformers thing up that I didn't want to break my flow and create a video so that's that's how things function you got to make trade-offs sometimes and when we are at this point of talking about time management let me let me let me walk you through how my like commit history looked like and how my how I organized my time during this three weeks okay so let me let me walk you through my schedule because I mean everybody's like I've noticed a lot of comments people thinking everybody is so much faster than they are and that's so much so not true it just takes so much time and let me let me walk you through my commit history if I open this up and let me find some representative day like November 5th and okay so let's start like I usually start my day around around like 929 30 a.m. and so I code until maybe 11 15 so usually around two hours every morning and then what I do I take a short break and like take a stroll get something to eat and I start working at Microsoft where I also work full-time so it's a really tight schedule and after I finish my my daily daily work at Microsoft where so the good thing about Microsoft and that's really awesome is that it's really it's a truly like a meritocracy system so my managers he doesn't care if I'm like working 10 hours or or six hours he cares that I get my done and that's this super rewarding I'll get fired because of this video I'm joking so so basically I do work around eight hours but the good thing is they value what you do and how productive you are and how much you sit down in that chair doesn't matter you get done and that's it so after I finish my my daily work I like usually take a power nap and depending how tired I am I maybe whatever I take some more more time to do like rest and then around usually around like 9 9 p.m. like 830 whatever I get working I get to to to coding again and so I work until maybe 11 p.m. 1130 so that's again like two three hours of quality coding so that's that's a that a that thing accumulates so that's like four or five hours every single work day and then I have weekends where I'm even more productive because I don't have to work at Microsoft on weekends and at the end of the day during work days I was usually either like I usually take some time to chill I either watch like South Park to just get my like just relax my brain or recently over the last week I've been just watching Lex Friedman's podcast and there is just so many interesting people there and just kind of also relax me even though I'm learning by watching that podcast as well so that's that's cool and now if somebody told me like even two years ago like about this schedule I would tell like I would tell that person like you're like nuts and the thing is depending on your mindset and where like your phase of life and like your situation this may not sound feasible but like that's what I thought like two years ago and then things change and it's totally fine so don't feel any pressure for not working as hard as somebody else because everybody's got their own pace and that's that's super fine like it took me 12 years to get in a working to acquire working habits that I have right now and initially I was just like learning different stuff on my own and structuring my journeys but they were not related to programming whatsoever so I was like learning about human languages I'm really passionate that was my of my there was one of my main hobbies when I was in high school and beginning of my faculty I was training bunch of sports I tried so many sports out there so I was just exploring and only at 19 did I get exposed to to programming and that's totally fine you can still achieve things even when you start a bit later like when you're 19 that's that's super totally fine like one more thing and I know it's kind of cliche but like life's not fair so like Bill Gates back in the 60s had a computer when he was like 12 or 13 and I mean he started programming when he was 12 and it's still I'll tell you something it still doesn't matter you can start when you're 19 20 25 30 and it's still not late if you keep up the good work and you keep like being consistent whatever you do you'll get better and you'll achieve you'll get where you want to be and that's it I mean chillax and lastly there is one more thing I want to mention and that's failures and like turn your head around you and look at successful people around you and they all had failures and if they're honest enough they'll tell you about those if they're not well so I think this guy like he's a Stanford PhD and he was open enough and honest enough to share his his journey and that's how pretty much everybody's journey looks like you get a lot of failures in his case it's like paper rejections but then you have a couple of successes and everybody knows you for your successes and that's it and like me personally I had like a bunch of failures like I was rejected by Facebook I was rejected by Nvidia I was rejected by Microsoft and then my second attempt I got like accepted into Microsoft but even if was like third attempt or fourth attempt it doesn't matter like as long as you like keep learning and you keep applying that's that you're set up for success if you're just applying then you're just spamming people like you got to keep on learning and applying and wherever you want to be you'll get there like sometimes when you get rejected it doesn't have to be that you're bad like I don't know like hiring pipelines are in big companies in big tech companies are not perfect they're far from perfect enough about motivational speaking hopefully I just wanted to share my personal story and some of details I think many YouTube videos out there are doing similar things there's already so much high quality educational resources and there is too little personal stories that I felt like sharing some some of the details from my life and from my workflow and I hope you found this helpful if you did please leave a comment in this comment section because I'll start creating more of these if you if you found useful okay all of that being said let me kind of explain you how I think about approaching a new problem and new project so a good thing about transformer project and in general about many software projects is that they can be modularized and what I mean by that is you can basically orthogonally I independently develop certain components and not care about others so this project was pretty much split into two parts so the first one takes care of training the model and then you have the second part where once you've trained the model you want to do you want to do translations and you want to do some decoding on those models so once once you split like the functionality is like that you can focus on pretty much basically only on the training side and I had only three files let's let's say three files three main files which I had to develop in order to get like the transformer model trained and the first one is obviously like the transformer model a pie file which contains the architecture the definition of the model itself and so the good thing is when I said it's orthogonal so you can basically I can I basically just developed this model and then I created a small main function inside of this Python file where I could test whether the model is working or not before I move ahead and develop the other parts of the project so it's never linear like that you usually jump a little bit here a little bit there but let's say I basically first developed this this model and then and then I went and did the other things so I figured out some bugs doing this which is a good thing better catch them sooner than later and I'll get into problems a bit later now let me just give you an overview of the project so the second thing is second important thing is like data and you want to load the data you want to load like in this case in this case so the transformer model I should mention that is is trained for the machine translation task so basically want to translate from one language which is called a source language into another language which is called a target language and the languages I use were English and German or German to English vice versa so I had lots of problems with loading data in in pytorch first of all because I was usually I'm used to computer vision projects and this was natural language processing so it was kind of a little bit of paradigm shift and second of all it's nowhere as near as near as good as historic vision is for computer vision so I stumbled across two significant problems first one was like functionality wise so the thing didn't work as I expected to do like a bucket iterate something called bucket iterators which were supposed to batch the data in an optimal way weren't working as I was expecting them to and the second problem is the performance problem so the like tokenization procedure and everything took a lot of time like every time I wanted to run the training loop it took like 45 seconds and it's so annoying and I kind I think I went from no I went from 66 seconds to like two and a half seconds which was a massive optimization jump and they helped me through it much faster and that's super important in deep learning and machine learning in general like you want to have a like a really short iteration cycle that that's that's that helps you build stuff better and faster so I'll tell you about some of the problems I had developing this like the data pipeline but first as I mentioned let me just quickly go through other files so then we have like the training loop and basically the training loop is pretty simple once you take a look so let me let me show you so basically yeah you I load them I prepare the model I prepare the data then what I do is I just prepare some things like for label smoothing I won't get into details what that is like optimizers and then the loop is really simple so for a number of epochs I use 20 because just looking at the the attention is all you need paper and calculating stuff you can see they approximately had 19 epochs on the WMT 14 data set so that's what I did also and I didn't get into any situation regions until 20 20 bucks it was totally fine and so the tration loop so I just I basically just have so I do the training loop and then I do the validation loop because you want to see how the model is generalizing to data that he hasn't seen during the training so training loop is pretty so I'm already getting into too much details so you basically once I said a training loop I can I can I can run the whole thing I can test it and then what I did is I quickly made a translation script which would test whether the model was doing what it's supposed to be doing and that's like translating from language to language and that's super important like you want to have like end-to-end system as soon as possible and then you can keep on iterating and improving either the model or the data loading and that's how I did it so I first had the whole thing working end-to-end and only then I did those optimizations etc I also had to develop decoding algorithms so what I did is I developed a simple one that's called like greedy decoding where basically so once you get the like the output distribution for output distribution for us for the next token you basically figure out where the highest probability is and that position corresponds to a certain token in your vocabulary so basically you find the next token in your output or target sentence in that way and greedy is the simplest method what they used in the paper was actually something called beam search where you keep like a like a like a number of hypotheses running in parallel and you at the end you're left with the with the with the hypothesis that's like most probable and you keep that one and you output that one as the output sentence finally I also have something called playground file and I do that not only because I'm open sourcing this and treating this like as a learning resource for others but it's also a really good thing for me because it helped me understand something a lot a lot better and because it's much easier I'm a highly visual person so I like to visualize stuff so for example there is this thing called if you're familiar with the transformer model they have something called positional encodings which you basically add to tokens embedding vectors and so they the formula is kind of complicated and then once you visualize it it's not as complicated so here is how it looks like for like so you basically take one row from this image and so that this like a vector of numbers and you add that to to the token at this particular position so row zero would be added to token representations at position zero or one would go to token representations at position one etc so that was the like the high-level overview of how I think about the project when I'm developing the project so I like to separate functionalities I like to get end-to-end system working as soon as possible and then I kind of go into depth so there's the same way I learned theory I always start high level so first like create a skeleton and then start filling in the gaps this the same I pretty much do the same thing I use the same strategy when I develop like software projects so that was the high-level overview of the project now just one I want to give you like a couple I want to like kind of tell you about a couple of problems I struggle with so one of them is when I was developing the the transformer model itself so it's always a good thing to kind of print out the layers you have in the model to print out the shapes and in general just to see how many parameters the model has so doing that I figured out that my multi had attention was actually I was I was referencing the same object in memory in multiple encoder layers and there was simply a bug which I discovered because I was I printed out these like layer layer names and I saw there there were some like multi had attention layers missing I also so basically if I change this to true here because the the paper explained they they had the base model and they had the big model so if I put the big one and I run the script I'll get I'm printing out the number of parameters and you can see here like 176 million parameters and that's also a good thing to do because I knew that the big model had 175 million parameters so this kind of confirmed that I'm that I have the same number of parameters as the paper stated so those are just some of the things you want to do because you you want to make sure that something is correct it's kind of it's a kind of a test pretty much yeah so the second fun experience I had was during when I was developing beam search functionality and I actually still haven't finished it yet I have really tested but beam is still not working and while I was kind of investigating how other people did it I found like in tensor to tensor tensor to tensor library I found like a line like this like length penalty equals to TF power of 5 plus something divided by 6 part to the alpha raised to the alpha power and was like what the so the thing is I beg you if you ever write code in now like a public open source library please leave a comment you will make like live so much easier so I don't know how but I like kind of went through the code and some other people's codes like open it and empty etc and I figured out there is a paper which introduced this is a terry heuristic and it kind of just works I mean and you can see here 5 plus blah blah raised to the power of alpha I was like what the fuck so those are just I just wanted to give you a glimpse of things you you encounter where you while you're working on these projects it's it's super interesting super funny stuff you can you can see okay this video is getting already a bit too long let me just show you one more interesting problem I stumbled upon while working on the transformer project so it has to do with Torch text and loading data and basically you have this class in Torch text which is called bucket iterator and it's something similar to data loader if you if you're familiar with computer vision like Torch vision library and what happened basically is that let me see if I can find so so what happened here is that like the documentation said something totally different from the results I was getting so basically what the bucket iterator said is minimizes amount of padding needed while producing freshly shuffled batches for each new epoch and that's not true like I had to dig so deep into the source code and finally I submitted like I saw there is already some issue pending and I just commented and like kind of said yeah that guy had right he stumbled upon the same bug as I did and basically you have to set this sort within batch to true otherwise you won't get like bad padding minimized and once you do that everything so so then you get the functionality you actually wanted and then there was a second thing so what I wanted to do is because certain sentences are much longer than other so if you if you just set the batch number as a fixed number so for example always load eight sentences sometimes the batch will be huge sometimes it will be a really really like small and that's a bad thing because you want to maximize the amount of VRAM on your GPU that you're using so what I had to do is I had to to write down this custom function which will which will basically make sure that I always have like the same number of approximately the same number of tokens in the batch and the annotated transformer luckily had something similar implemented although they didn't comment why this was like why this was there and so I struggled a lot until I actually understood the source code and then I figured everything made sense so now hopefully if you go through my code it will be a bit easier to understand what happened here just one more quick thing that's that I had to add this basically custom data set because using space tokenizers at the moment it took me 66 seconds 70 seconds whatever to to re-tokenize the the source and target sentences every single time and so I just decided why not tokenized once save it as a like simple txt file and then instead of re-tokenizing things again just load the txt file and that now takes like two and a half seconds so in a nutshell that's what I did with these data set wrapper and this fast translation data set wrapper class so that was a short story about like some problems I encountered now like for the end let me just show you how the thing functions and how the translation script actually works and I think it's really interesting and magical for me because I was usually doing computer vision things and then once you see this thing actually working and translating from English to German and vice versa it's pretty fascinating if you ask me so I've got up so you basically just have to input the sentence depending so I have an English sentence how are you doing today whatever like set as a source sentence and we have the model that was trained on IWSLT data set and it's translating from English to German so the name is pretty indicative of what the model is doing and where the model was trained on so you just have to keep these two in sync with the model so if you change the model to G to E German to English you will just have to set the enum here to G to E instead of E to G that's it once I start this thing let me run it it takes a little bit of time to half seconds to load the the like the the the vocabularies and what cat to load vocabulary and then it'll just translate why is it not working okay here it is it took two and a two point six seconds to load the data and then you can see the input sentence the source sentence tokenized and you see the output sentence we get the scene heute which means the same thing pretty much although enum is a German word like a polite way to say you so there is nothing stopping me from translating from German to English let me just change this thing to German to English let me change the model to German to English and now if I input for example we get this dia which translates as yeah how are you basically in English let me see what the model will say how the model would translate this this time and it basically translated this as how are you which is pretty much something called gold translation which means like ground truth translation this looks pretty pretty decent the model is not perfect you can find some sentences which would kind of confuse the model so even a simple sentence like a spin I'm berlinna which is a sentence like I think Kennedy said when he was back in Berlin back in the days and it means I'm a Berliner I'm a I'm person living in Berlin and if I input this and run it so the model will actually the thing is the model doesn't have a token Berliner inside the vocabulary so that's why it will say Berlin instead of say I mean I'm anthropomorphizing the model here so it will output Berlin so I am a Berlin so if the model had Berliner it would output Berliner so that's another another point I want to make here so if I had something called byte pairing coding I could the model would be much more expressive in creating these words then at this point of time but that's a future work and I'll edit hopefully like maybe next week or something so there are many other things I could show you but for the sake of this video not being like one hour long let me just show you how the perf what amount of hardware requirements do you need in order to run translations not in order to train the model you would need a lot of GPUs to train the model actually had have some comments in the readme please go go ahead and read that part but for like just for for translations let me just open the task manager and if I open it up if I run the script now we can follow how much GPU is needed to to get the translation so it's so it's basically only a small spike so you won't need as much hardware like like a really fancy GPU if you're just running translations on the other side if you're training this model you'll need a lot of power so that that pretty much wraps it up hope you like this video there was a lot of information in here so what my next steps will be is I'll start reading research papers again I want to explore what state-of-the-art in natural language processing now in 2020 so I'll be spending like maybe reading one paper in the morning one paper in the evening that amounts to maybe at least ten papers a week which is good enough to get me like familiar with the field in like 10 days period two weeks so yeah that was pretty much it if you like this video go ahead and subscribe to this channel and hit the bell icon to get notified when I upload a new video and until next time keep learning
[{"start": 0.0, "end": 5.58, "text": " Finally, finally I've open sourced my implementation of the original"}, {"start": 5.58, "end": 10.86, "text": " transformer paper, Attention is All You Need, and it took me like three weeks of"}, {"start": 10.86, "end": 16.580000000000002, "text": " non-stop working on this and I'm super happy. I learned a lot and hopefully"}, {"start": 16.580000000000002, "end": 21.54, "text": " you'll find it as a valuable resource as well. So yeah, let me just walk you"}, {"start": 21.54, "end": 27.54, "text": " through this, read me. This is the architecture. I have some nice"}, {"start": 27.54, "end": 32.32, "text": " visualizations here and I have two models which I've"}, {"start": 32.32, "end": 40.12, "text": " pre-trained, which I've linked. So yeah, before I get into this, like the"}, {"start": 40.12, "end": 44.36, "text": " code itself, I just want to tell you about a couple of things, like the whole"}, {"start": 44.36, "end": 49.879999999999995, "text": " journey I had learning about this. And so those of you who've been following me"}, {"start": 49.879999999999995, "end": 56.2, "text": " over the last like a couple of weeks, you know that I've been... so I've been"}, {"start": 56.2, "end": 61.0, "text": " working on transformers for more than a month now and I first started by like"}, {"start": 61.0, "end": 66.64, "text": " just reading about theory, reading like research papers for more than like two"}, {"start": 66.64, "end": 72.56, "text": " plus weeks. And then after I finished that one, I basically started coding the"}, {"start": 72.56, "end": 76.88, "text": " implementation from scratch and that took me around three weeks more or less."}, {"start": 76.88, "end": 82.88, "text": " So let me start with why I did it. So the thing is I looked for like good"}, {"start": 82.88, "end": 87.52, "text": " resources to understand transformers but not on the like level of theory but like"}, {"start": 87.52, "end": 91.96, "text": " to be able to actually understand the code itself. And I basically only found"}, {"start": 91.96, "end": 98.6, "text": " two decent resources, good resources. Basically, I'm not counting in Hugging"}, {"start": 98.6, "end": 103.08, "text": " Face because they do have a really awesome library of transformers but they"}, {"start": 103.08, "end": 108.38, "text": " are like more for like as a black box experience type of usage and not for"}, {"start": 108.38, "end": 112.28, "text": " going through the code and understanding how stuff works. So the two resources I"}, {"start": 112.28, "end": 117.0, "text": " mentioned are the annotated transformer is the first one and then PyTorch"}, {"start": 117.0, "end": 121.32000000000001, "text": " official implementation. Now the thing with PyTorch official implementation is"}, {"start": 121.32000000000001, "end": 126.36, "text": " that like their multi-headed attention module is awesome, like functionality"}, {"start": 126.36, "end": 131.28, "text": " wise it works. But the thing is it's so generic because they got to cover a"}, {"start": 131.28, "end": 135.48, "text": " bunch of use cases and so it's so complicated to actually go through the"}, {"start": 135.48, "end": 140.72, "text": " code and understand how it works. So my idea was to just focus on this specific"}, {"start": 140.72, "end": 148.08, "text": " use case and like of this transformer model from 2017 and that's it. On the"}, {"start": 148.08, "end": 155.48, "text": " other hand, the annotated transformer block is basically suited only for"}, {"start": 155.48, "end": 161.16, "text": " researchers and also had a couple of bugs. It's a it's an overall really nice"}, {"start": 161.16, "end": 166.92, "text": " resource and helped me a lot but it's really only for researchers and they"}, {"start": 166.92, "end": 173.16, "text": " also like kind of suppose you understand the source code of Torch text which is"}, {"start": 173.16, "end": 179.39999999999998, "text": " PyTorch's like module for manipulating NLP tasks which is usually not the"}, {"start": 179.39999999999998, "end": 184.2, "text": " not the case right. So I want to I want to make a couple of points before we go"}, {"start": 184.2, "end": 188.44, "text": " and take an overview of the code and I explain you how I struggled, which"}, {"start": 188.44, "end": 192.27999999999997, "text": " problems I had, how I approached it. I won't get into every single detail"}, {"start": 192.28, "end": 197.04, "text": " because the code base is now already pretty big. I'll show you some details but"}, {"start": 197.04, "end": 203.28, "text": " before that I want to tell you that making good projects takes time. So I had"}, {"start": 203.28, "end": 207.28, "text": " I had a couple of comments. I had Rahul commenting, thanks for telling us that"}, {"start": 207.28, "end": 211.12, "text": " understanding and coding a new AI algorithm takes a few months. I always"}, {"start": 211.12, "end": 215.48, "text": " felt like people are going too fast and doing things in just days and I cannot"}, {"start": 215.48, "end": 220.6, "text": " keep up. But thank you, it gives me motivation. So thank you Rahul. The thing"}, {"start": 220.6, "end": 226.28, "text": " is I mean it's funny like my first so my first attempt at trying to"}, {"start": 226.28, "end": 230.35999999999999, "text": " reconstruct a research paper took three months. So I was coding three months"}, {"start": 230.35999999999999, "end": 236.2, "text": " non-stop and the reason being the authors did not open source their like"}, {"start": 236.2, "end": 239.76, "text": " implementation and you couldn't find the code wasn't as popular as the"}, {"start": 239.76, "end": 244.68, "text": " transformer so I couldn't find snippets of code anywhere. So what I what I ended"}, {"start": 244.68, "end": 248.79999999999998, "text": " up doing is just like trying and reconstructing the the the paper by"}, {"start": 248.8, "end": 254.28, "text": " just reading text which is super hard and it takes time and I had lots of back"}, {"start": 254.28, "end": 261.24, "text": " and forth with the authors to understand certain big details and it's like so the"}, {"start": 261.24, "end": 268.28000000000003, "text": " point I want to make here is don't fall for that learn topic huge topic X in"}, {"start": 268.28000000000003, "end": 274.16, "text": " five days. I mean that doesn't work like that and I'm just wondering I think I"}, {"start": 274.16, "end": 277.76, "text": " know why that's the case like you have you have Siraj Rawal who had videos"}, {"start": 277.76, "end": 284.24, "text": " like learn tensorflow in five minutes and I mean if you I want to tell you"}, {"start": 284.24, "end": 287.88, "text": " something I mean if you if you believe that crap you're in a really bad"}, {"start": 287.88, "end": 292.56, "text": " position because that's going to set you up for failure and why because you're"}, {"start": 292.56, "end": 297.03999999999996, "text": " basically it's basically setting up setting you up for impatience whereas"}, {"start": 297.03999999999996, "end": 302.4, "text": " you really truly need patience in order to create anything great like every"}, {"start": 302.4, "end": 306.36, "text": " single project like the projects I'm doing at Microsoft they take that's like"}, {"start": 306.36, "end": 312.12, "text": " a multi like multi like team effort that lasts at least six months one year two"}, {"start": 312.12, "end": 316.64, "text": " years three years so anything significant takes time in artificial"}, {"start": 316.64, "end": 320.48, "text": " intelligence and computer science whatever the topic it is like I have"}, {"start": 320.48, "end": 325.26, "text": " skipped creating a video last week last weekend because I was so in a flow of"}, {"start": 325.26, "end": 331.0, "text": " cutting this transformers thing up that I didn't want to break my flow and create"}, {"start": 331.0, "end": 334.6, "text": " a video so that's that's how things function you got to make trade-offs"}, {"start": 334.6, "end": 338.96000000000004, "text": " sometimes and when we are at this point of talking about time management let me"}, {"start": 338.96000000000004, "end": 343.16, "text": " let me let me walk you through how my like commit history looked like and how"}, {"start": 343.16, "end": 347.40000000000003, "text": " my how I organized my time during this three weeks okay so let me let me walk"}, {"start": 347.40000000000003, "end": 351.76000000000005, "text": " you through my schedule because I mean everybody's like I've noticed a lot of"}, {"start": 351.76000000000005, "end": 357.58000000000004, "text": " comments people thinking everybody is so much faster than they are and that's so"}, {"start": 357.58000000000004, "end": 360.92, "text": " much so not true it just takes so much time and let me let me walk you through"}, {"start": 360.92, "end": 365.8, "text": " my commit history if I open this up and let me find some representative day like"}, {"start": 365.8, "end": 373.16, "text": " November 5th and okay so let's start like I usually start my day around around"}, {"start": 373.16, "end": 383.32, "text": " like 929 30 a.m. and so I code until maybe 11 15 so usually around two hours"}, {"start": 383.32, "end": 389.96000000000004, "text": " every morning and then what I do I take a short break and like take a stroll get"}, {"start": 389.96, "end": 395.12, "text": " something to eat and I start working at Microsoft where I also work full-time so"}, {"start": 395.12, "end": 401.71999999999997, "text": " it's a really tight schedule and after I finish my my daily daily work at"}, {"start": 401.71999999999997, "end": 406.12, "text": " Microsoft where so the good thing about Microsoft and that's really awesome is"}, {"start": 406.12, "end": 411.67999999999995, "text": " that it's really it's a truly like a meritocracy system so my managers he"}, {"start": 411.67999999999995, "end": 417.2, "text": " doesn't care if I'm like working 10 hours or or six hours he cares that I"}, {"start": 417.2, "end": 424.52, "text": " get my done and that's this super rewarding I'll get fired because of this"}, {"start": 424.52, "end": 430.28, "text": " video I'm joking so so basically I do work around eight hours but the good"}, {"start": 430.28, "end": 434.98, "text": " thing is they value what you do and how productive you are and how much you sit"}, {"start": 434.98, "end": 440.32, "text": " down in that chair doesn't matter you get done and that's it so after I finish"}, {"start": 440.32, "end": 444.64, "text": " my my daily work I like usually take a power nap and depending how tired I am I"}, {"start": 444.64, "end": 450.28, "text": " maybe whatever I take some more more time to do like rest and then around"}, {"start": 450.28, "end": 457.88, "text": " usually around like 9 9 p.m. like 830 whatever I get working I get to to to"}, {"start": 457.88, "end": 464.52, "text": " coding again and so I work until maybe 11 p.m. 1130 so that's again like two"}, {"start": 464.52, "end": 469.64, "text": " three hours of quality coding so that's that's a that a that thing accumulates"}, {"start": 469.64, "end": 474.44, "text": " so that's like four or five hours every single work day and then I have weekends"}, {"start": 474.44, "end": 477.84, "text": " where I'm even more productive because I don't have to work at Microsoft on"}, {"start": 477.84, "end": 482.47999999999996, "text": " weekends and at the end of the day during work days I was usually either"}, {"start": 482.47999999999996, "end": 488.0, "text": " like I usually take some time to chill I either watch like South Park to just get"}, {"start": 488.0, "end": 493.32, "text": " my like just relax my brain or recently over the last week I've been just"}, {"start": 493.32, "end": 497.4, "text": " watching Lex Friedman's podcast and there is just so many interesting people"}, {"start": 497.4, "end": 502.0, "text": " there and just kind of also relax me even though I'm learning by watching"}, {"start": 502.0, "end": 506.4, "text": " that podcast as well so that's that's cool and now if somebody told me like"}, {"start": 506.4, "end": 513.42, "text": " even two years ago like about this schedule I would tell like I would tell"}, {"start": 513.42, "end": 518.72, "text": " that person like you're like nuts and the thing is depending on your mindset"}, {"start": 518.72, "end": 524.36, "text": " and where like your phase of life and like your situation this may not sound"}, {"start": 524.36, "end": 528.88, "text": " feasible but like that's what I thought like two years ago and then things"}, {"start": 528.88, "end": 533.92, "text": " change and it's totally fine so don't feel any pressure for not working as"}, {"start": 533.92, "end": 537.44, "text": " hard as somebody else because everybody's got their own pace and that's"}, {"start": 537.44, "end": 544.16, "text": " that's super fine like it took me 12 years to get in a working to acquire"}, {"start": 544.16, "end": 548.04, "text": " working habits that I have right now and initially I was just like learning"}, {"start": 548.04, "end": 551.6, "text": " different stuff on my own and structuring my journeys but they were"}, {"start": 551.6, "end": 555.44, "text": " not related to programming whatsoever so I was like learning about human"}, {"start": 555.44, "end": 560.0, "text": " languages I'm really passionate that was my of my there was one of my main"}, {"start": 560.0, "end": 563.6, "text": " hobbies when I was in high school and beginning of my faculty I was training"}, {"start": 563.6, "end": 568.88, "text": " bunch of sports I tried so many sports out there so I was just exploring and"}, {"start": 568.88, "end": 574.52, "text": " only at 19 did I get exposed to to programming and that's totally fine you"}, {"start": 574.52, "end": 578.36, "text": " can still achieve things even when you start a bit later like when you're 19"}, {"start": 578.36, "end": 581.84, "text": " that's that's super totally fine like one more thing and I know it's kind of"}, {"start": 581.84, "end": 587.0, "text": " cliche but like life's not fair so like Bill Gates back in the 60s had a"}, {"start": 587.0, "end": 593.48, "text": " computer when he was like 12 or 13 and I mean he started programming when he was"}, {"start": 593.48, "end": 598.32, "text": " 12 and it's still I'll tell you something it still doesn't matter you can"}, {"start": 598.32, "end": 603.4, "text": " start when you're 19 20 25 30 and it's still not late if you keep up the"}, {"start": 603.4, "end": 608.4399999999999, "text": " good work and you keep like being consistent whatever you do you'll get"}, {"start": 608.4399999999999, "end": 612.16, "text": " better and you'll achieve you'll get where you want to be and that's it I"}, {"start": 612.16, "end": 616.16, "text": " mean chillax and lastly there is one more thing I want to mention and that's"}, {"start": 616.16, "end": 621.28, "text": " failures and like turn your head around you and look at successful people around"}, {"start": 621.28, "end": 625.68, "text": " you and they all had failures and if they're honest enough they'll tell you"}, {"start": 625.68, "end": 630.16, "text": " about those if they're not well so I think this guy like he's a Stanford PhD"}, {"start": 630.16, "end": 637.04, "text": " and he was open enough and honest enough to share his his journey and that's how"}, {"start": 637.04, "end": 642.28, "text": " pretty much everybody's journey looks like you get a lot of failures in his"}, {"start": 642.28, "end": 646.56, "text": " case it's like paper rejections but then you have a couple of successes and"}, {"start": 646.56, "end": 651.04, "text": " everybody knows you for your successes and that's it and like me personally I"}, {"start": 651.04, "end": 655.48, "text": " had like a bunch of failures like I was rejected by Facebook I was rejected by"}, {"start": 655.48, "end": 660.9200000000001, "text": " Nvidia I was rejected by Microsoft and then my second attempt I got like"}, {"start": 660.9200000000001, "end": 665.4, "text": " accepted into Microsoft but even if was like third attempt or fourth attempt it"}, {"start": 665.4, "end": 670.48, "text": " doesn't matter like as long as you like keep learning and you keep applying"}, {"start": 670.48, "end": 675.04, "text": " that's that you're set up for success if you're just applying then you're just"}, {"start": 675.04, "end": 678.88, "text": " spamming people like you got to keep on learning and applying and wherever you"}, {"start": 678.88, "end": 682.64, "text": " want to be you'll get there like sometimes when you get rejected it"}, {"start": 682.64, "end": 686.48, "text": " doesn't have to be that you're bad like I don't know like hiring pipelines are"}, {"start": 686.48, "end": 691.16, "text": " in big companies in big tech companies are not perfect they're far from perfect"}, {"start": 691.16, "end": 695.72, "text": " enough about motivational speaking hopefully I just wanted to share my"}, {"start": 695.72, "end": 700.16, "text": " personal story and some of details I think many YouTube videos out there are"}, {"start": 700.16, "end": 704.96, "text": " doing similar things there's already so much high quality educational resources"}, {"start": 704.96, "end": 709.96, "text": " and there is too little personal stories that I felt like sharing some some of"}, {"start": 709.96, "end": 714.64, "text": " the details from my life and from my workflow and I hope you found this"}, {"start": 714.64, "end": 719.96, "text": " helpful if you did please leave a comment in this comment section because"}, {"start": 719.96, "end": 725.24, "text": " I'll start creating more of these if you if you found useful okay all of that"}, {"start": 725.24, "end": 731.4000000000001, "text": " being said let me kind of explain you how I think about approaching a new"}, {"start": 731.4000000000001, "end": 736.8000000000001, "text": " problem and new project so a good thing about transformer project and in general"}, {"start": 736.8, "end": 741.56, "text": " about many software projects is that they can be modularized and what I mean"}, {"start": 741.56, "end": 745.56, "text": " by that is you can basically orthogonally I independently develop"}, {"start": 745.56, "end": 751.16, "text": " certain components and not care about others so this project was pretty much"}, {"start": 751.16, "end": 756.64, "text": " split into two parts so the first one takes care of training the model and"}, {"start": 756.64, "end": 761.5999999999999, "text": " then you have the second part where once you've trained the model you want to do"}, {"start": 761.6, "end": 769.8000000000001, "text": " you want to do translations and you want to do some decoding on those models so"}, {"start": 769.8000000000001, "end": 773.6800000000001, "text": " once once you split like the functionality is like that you can focus"}, {"start": 773.6800000000001, "end": 778.48, "text": " on pretty much basically only on the training side and I had only three"}, {"start": 778.48, "end": 784.48, "text": " files let's let's say three files three main files which I had to develop in"}, {"start": 784.48, "end": 790.36, "text": " order to get like the transformer model trained and the first one is obviously"}, {"start": 790.36, "end": 795.5600000000001, "text": " like the transformer model a pie file which contains the architecture the"}, {"start": 795.5600000000001, "end": 800.92, "text": " definition of the model itself and so the good thing is when I said it's"}, {"start": 800.92, "end": 807.0, "text": " orthogonal so you can basically I can I basically just developed this model and"}, {"start": 807.0, "end": 812.32, "text": " then I created a small main function inside of this Python file where I could"}, {"start": 812.32, "end": 818.2, "text": " test whether the model is working or not before I move ahead and develop the"}, {"start": 818.2, "end": 822.5600000000001, "text": " other parts of the project so it's never linear like that you usually jump a"}, {"start": 822.5600000000001, "end": 826.5600000000001, "text": " little bit here a little bit there but let's say I basically first developed"}, {"start": 826.5600000000001, "end": 831.9200000000001, "text": " this this model and then and then I went and did the other things so I figured"}, {"start": 831.9200000000001, "end": 836.6, "text": " out some bugs doing this which is a good thing better catch them sooner than"}, {"start": 836.6, "end": 841.8000000000001, "text": " later and I'll get into problems a bit later now let me just give you an"}, {"start": 841.8000000000001, "end": 846.6800000000001, "text": " overview of the project so the second thing is second important thing is like"}, {"start": 846.68, "end": 851.0, "text": " data and you want to load the data you want to load like in this case in this"}, {"start": 851.0, "end": 855.04, "text": " case so the transformer model I should mention that is is trained for the"}, {"start": 855.04, "end": 859.3599999999999, "text": " machine translation task so basically want to translate from one language which"}, {"start": 859.3599999999999, "end": 862.92, "text": " is called a source language into another language which is called a target"}, {"start": 862.92, "end": 867.5999999999999, "text": " language and the languages I use were English and German or German to English"}, {"start": 867.5999999999999, "end": 874.64, "text": " vice versa so I had lots of problems with loading data in in pytorch first of"}, {"start": 874.64, "end": 878.92, "text": " all because I was usually I'm used to computer vision projects and this was"}, {"start": 878.92, "end": 882.6, "text": " natural language processing so it was kind of a little bit of paradigm shift"}, {"start": 882.6, "end": 888.48, "text": " and second of all it's nowhere as near as near as good as historic vision is for"}, {"start": 888.48, "end": 894.52, "text": " computer vision so I stumbled across two significant problems first one was like"}, {"start": 894.52, "end": 899.84, "text": " functionality wise so the thing didn't work as I expected to do like a bucket"}, {"start": 899.84, "end": 903.68, "text": " iterate something called bucket iterators which were supposed to batch"}, {"start": 903.68, "end": 907.5999999999999, "text": " the data in an optimal way weren't working as I was expecting them to and"}, {"start": 907.5999999999999, "end": 912.68, "text": " the second problem is the performance problem so the like tokenization"}, {"start": 912.68, "end": 917.16, "text": " procedure and everything took a lot of time like every time I wanted to run"}, {"start": 917.16, "end": 922.3199999999999, "text": " the training loop it took like 45 seconds and it's so annoying and I kind"}, {"start": 922.3199999999999, "end": 927.5999999999999, "text": " I think I went from no I went from 66 seconds to like two and a half seconds"}, {"start": 927.5999999999999, "end": 933.16, "text": " which was a massive optimization jump and they helped me through it much"}, {"start": 933.16, "end": 936.0799999999999, "text": " faster and that's super important in deep learning and machine learning in"}, {"start": 936.0799999999999, "end": 940.68, "text": " general like you want to have a like a really short iteration cycle that that's"}, {"start": 940.68, "end": 948.28, "text": " that's that helps you build stuff better and faster so I'll tell you about some"}, {"start": 948.28, "end": 955.4399999999999, "text": " of the problems I had developing this like the data pipeline but first as I"}, {"start": 955.4399999999999, "end": 959.92, "text": " mentioned let me just quickly go through other files so then we have like the"}, {"start": 959.92, "end": 965.4399999999999, "text": " training loop and basically the training loop is pretty simple once you take a"}, {"start": 965.4399999999999, "end": 971.4799999999999, "text": " look so let me let me show you so basically yeah you I load them I prepare"}, {"start": 971.4799999999999, "end": 977.76, "text": " the model I prepare the data then what I do is I just prepare some things like"}, {"start": 977.76, "end": 983.0, "text": " for label smoothing I won't get into details what that is like optimizers and"}, {"start": 983.0, "end": 988.12, "text": " then the loop is really simple so for a number of epochs I use 20 because just"}, {"start": 988.12, "end": 993.16, "text": " looking at the the attention is all you need paper and calculating stuff you can"}, {"start": 993.16, "end": 999.72, "text": " see they approximately had 19 epochs on the WMT 14 data set so that's what I did"}, {"start": 999.72, "end": 1005.12, "text": " also and I didn't get into any situation regions until 20 20 bucks it was totally"}, {"start": 1005.12, "end": 1011.08, "text": " fine and so the tration loop so I just I basically just have so I do the"}, {"start": 1011.08, "end": 1013.84, "text": " training loop and then I do the validation loop because you want to see"}, {"start": 1013.84, "end": 1018.72, "text": " how the model is generalizing to data that he hasn't seen during the training"}, {"start": 1018.72, "end": 1024.2, "text": " so training loop is pretty so I'm already getting into too much details so"}, {"start": 1024.2, "end": 1028.68, "text": " you basically once I said a training loop I can I can I can run the whole"}, {"start": 1028.68, "end": 1035.4, "text": " thing I can test it and then what I did is I quickly made a translation script"}, {"start": 1035.4, "end": 1039.76, "text": " which would test whether the model was doing what it's supposed to be doing and"}, {"start": 1039.76, "end": 1044.92, "text": " that's like translating from language to language and that's super important like"}, {"start": 1044.92, "end": 1050.52, "text": " you want to have like end-to-end system as soon as possible and then you can"}, {"start": 1050.52, "end": 1055.04, "text": " keep on iterating and improving either the model or the data loading and that's"}, {"start": 1055.04, "end": 1059.36, "text": " how I did it so I first had the whole thing working end-to-end and only then I"}, {"start": 1059.36, "end": 1064.4, "text": " did those optimizations etc I also had to develop decoding algorithms so what I"}, {"start": 1064.4, "end": 1069.0, "text": " did is I developed a simple one that's called like greedy decoding where"}, {"start": 1069.0, "end": 1073.8, "text": " basically so once you get the like the output distribution for output"}, {"start": 1073.8, "end": 1078.44, "text": " distribution for us for the next token you basically figure out where the"}, {"start": 1078.44, "end": 1082.96, "text": " highest probability is and that position corresponds to a certain token in your"}, {"start": 1082.96, "end": 1089.1, "text": " vocabulary so basically you find the next token in your output or target"}, {"start": 1089.1, "end": 1095.76, "text": " sentence in that way and greedy is the simplest method what they used in the"}, {"start": 1095.76, "end": 1101.04, "text": " paper was actually something called beam search where you keep like a like a like"}, {"start": 1101.04, "end": 1106.04, "text": " a number of hypotheses running in parallel and you at the end you're left"}, {"start": 1106.04, "end": 1110.24, "text": " with the with the with the hypothesis that's like most probable and you keep"}, {"start": 1110.24, "end": 1116.32, "text": " that one and you output that one as the output sentence finally I also have"}, {"start": 1116.32, "end": 1120.2, "text": " something called playground file and I do that not only because I'm open"}, {"start": 1120.2, "end": 1125.4, "text": " sourcing this and treating this like as a learning resource for others but it's"}, {"start": 1125.4, "end": 1130.24, "text": " also a really good thing for me because it helped me understand something a lot"}, {"start": 1130.24, "end": 1135.0, "text": " a lot better and because it's much easier I'm a highly visual person so I"}, {"start": 1135.0, "end": 1139.48, "text": " like to visualize stuff so for example there is this thing called if you're"}, {"start": 1139.48, "end": 1143.68, "text": " familiar with the transformer model they have something called positional"}, {"start": 1143.68, "end": 1150.92, "text": " encodings which you basically add to tokens embedding vectors and so they the"}, {"start": 1150.92, "end": 1154.52, "text": " formula is kind of complicated and then once you visualize it it's not as"}, {"start": 1154.52, "end": 1159.4, "text": " complicated so here is how it looks like for like so you basically take one row"}, {"start": 1159.4, "end": 1166.16, "text": " from this image and so that this like a vector of numbers and you add that to to"}, {"start": 1166.16, "end": 1170.92, "text": " the token at this particular position so row zero would be added to token"}, {"start": 1170.92, "end": 1176.8, "text": " representations at position zero or one would go to token representations at"}, {"start": 1176.8, "end": 1183.8, "text": " position one etc so that was the like the high-level overview of how I think"}, {"start": 1183.8, "end": 1187.6, "text": " about the project when I'm developing the project so I like to separate"}, {"start": 1187.6, "end": 1192.84, "text": " functionalities I like to get end-to-end system working as soon as possible and"}, {"start": 1192.84, "end": 1197.52, "text": " then I kind of go into depth so there's the same way I learned theory I always"}, {"start": 1197.52, "end": 1201.9199999999998, "text": " start high level so first like create a skeleton and then start filling in the"}, {"start": 1201.9199999999998, "end": 1206.48, "text": " gaps this the same I pretty much do the same thing I use the same strategy when"}, {"start": 1206.48, "end": 1211.32, "text": " I develop like software projects so that was the high-level overview of the"}, {"start": 1211.32, "end": 1216.3999999999999, "text": " project now just one I want to give you like a couple I want to like kind of"}, {"start": 1216.3999999999999, "end": 1221.3999999999999, "text": " tell you about a couple of problems I struggle with so one of them is when I"}, {"start": 1221.3999999999999, "end": 1226.9199999999998, "text": " was developing the the transformer model itself so it's always a good thing to"}, {"start": 1226.9199999999998, "end": 1232.96, "text": " kind of print out the layers you have in the model to print out the shapes and in"}, {"start": 1232.96, "end": 1237.52, "text": " general just to see how many parameters the model has so doing that I figured"}, {"start": 1237.52, "end": 1242.72, "text": " out that my multi had attention was actually I was I was referencing the"}, {"start": 1242.72, "end": 1248.08, "text": " same object in memory in multiple encoder layers and there was simply a"}, {"start": 1248.08, "end": 1253.36, "text": " bug which I discovered because I was I printed out these like layer layer names"}, {"start": 1253.36, "end": 1258.56, "text": " and I saw there there were some like multi had attention layers missing I"}, {"start": 1258.56, "end": 1265.6, "text": " also so basically if I change this to true here because the the paper explained"}, {"start": 1265.6, "end": 1269.36, "text": " they they had the base model and they had the big model so if I put the big"}, {"start": 1269.36, "end": 1274.0, "text": " one and I run the script I'll get I'm printing out the number of parameters"}, {"start": 1274.0, "end": 1280.04, "text": " and you can see here like 176 million parameters and that's also a good thing"}, {"start": 1280.04, "end": 1285.6799999999998, "text": " to do because I knew that the big model had 175 million parameters so this kind"}, {"start": 1285.6799999999998, "end": 1290.28, "text": " of confirmed that I'm that I have the same number of parameters as the paper"}, {"start": 1290.28, "end": 1295.56, "text": " stated so those are just some of the things you want to do because you you"}, {"start": 1295.56, "end": 1300.24, "text": " want to make sure that something is correct it's kind of it's a kind of a"}, {"start": 1300.24, "end": 1308.24, "text": " test pretty much yeah so the second fun experience I had was during when I was"}, {"start": 1308.24, "end": 1313.56, "text": " developing beam search functionality and I actually still haven't finished it yet"}, {"start": 1313.56, "end": 1319.52, "text": " I have really tested but beam is still not working and while I was kind of"}, {"start": 1319.52, "end": 1325.52, "text": " investigating how other people did it I found like in tensor to tensor tensor"}, {"start": 1325.52, "end": 1331.12, "text": " to tensor library I found like a line like this like length penalty equals to"}, {"start": 1331.12, "end": 1339.8, "text": " TF power of 5 plus something divided by 6 part to the alpha raised to the alpha"}, {"start": 1339.8, "end": 1346.68, "text": " power and was like what the so the thing is I beg you if you ever write code in"}, {"start": 1346.68, "end": 1352.48, "text": " now like a public open source library please leave a comment you will make"}, {"start": 1352.48, "end": 1358.5600000000002, "text": " like live so much easier so I don't know how but I like kind of went through the"}, {"start": 1358.5600000000002, "end": 1363.3600000000001, "text": " code and some other people's codes like open it and empty etc and I figured out"}, {"start": 1363.3600000000001, "end": 1370.1200000000001, "text": " there is a paper which introduced this is a terry heuristic and it kind of just"}, {"start": 1370.1200000000001, "end": 1375.68, "text": " works I mean and you can see here 5 plus blah blah raised to the power of alpha"}, {"start": 1375.68, "end": 1379.76, "text": " I was like what the fuck so those are just I just wanted to give you a glimpse"}, {"start": 1379.76, "end": 1384.52, "text": " of things you you encounter where you while you're working on these projects"}, {"start": 1384.52, "end": 1389.2, "text": " it's it's super interesting super funny stuff you can you can see okay this"}, {"start": 1389.2, "end": 1393.68, "text": " video is getting already a bit too long let me just show you one more interesting"}, {"start": 1393.68, "end": 1399.24, "text": " problem I stumbled upon while working on the transformer project so it has to do"}, {"start": 1399.24, "end": 1404.44, "text": " with Torch text and loading data and basically you have this class in Torch"}, {"start": 1404.44, "end": 1409.8, "text": " text which is called bucket iterator and it's something similar to data loader if"}, {"start": 1409.8, "end": 1414.72, "text": " you if you're familiar with computer vision like Torch vision library and"}, {"start": 1414.72, "end": 1421.8, "text": " what happened basically is that let me see if I can find so so what happened"}, {"start": 1421.8, "end": 1427.3600000000001, "text": " here is that like the documentation said something totally different from the"}, {"start": 1427.3600000000001, "end": 1432.96, "text": " results I was getting so basically what the bucket iterator said is minimizes"}, {"start": 1432.96, "end": 1437.48, "text": " amount of padding needed while producing freshly shuffled batches for each new"}, {"start": 1437.48, "end": 1444.3600000000001, "text": " epoch and that's not true like I had to dig so deep into the source code and"}, {"start": 1444.3600000000001, "end": 1450.48, "text": " finally I submitted like I saw there is already some issue pending and I just"}, {"start": 1450.48, "end": 1454.98, "text": " commented and like kind of said yeah that guy had right he stumbled upon the"}, {"start": 1454.98, "end": 1460.64, "text": " same bug as I did and basically you have to set this sort within batch to true"}, {"start": 1460.64, "end": 1467.3200000000002, "text": " otherwise you won't get like bad padding minimized and once you do that"}, {"start": 1467.3200000000002, "end": 1470.64, "text": " everything so so then you get the functionality you actually wanted and"}, {"start": 1470.64, "end": 1476.3200000000002, "text": " then there was a second thing so what I wanted to do is because certain sentences"}, {"start": 1476.3200000000002, "end": 1481.3200000000002, "text": " are much longer than other so if you if you just set the batch number as a fixed"}, {"start": 1481.3200000000002, "end": 1486.2800000000002, "text": " number so for example always load eight sentences sometimes the batch will be"}, {"start": 1486.28, "end": 1492.04, "text": " huge sometimes it will be a really really like small and that's a bad thing"}, {"start": 1492.04, "end": 1496.76, "text": " because you want to maximize the amount of VRAM on your GPU that you're using"}, {"start": 1496.76, "end": 1505.04, "text": " so what I had to do is I had to to write down this custom function which will"}, {"start": 1505.04, "end": 1511.36, "text": " which will basically make sure that I always have like the same number of"}, {"start": 1511.36, "end": 1516.0, "text": " approximately the same number of tokens in the batch and the annotated"}, {"start": 1516.0, "end": 1519.44, "text": " transformer luckily had something similar implemented although they didn't"}, {"start": 1519.44, "end": 1525.6, "text": " comment why this was like why this was there and so I struggled a lot"}, {"start": 1525.6, "end": 1529.52, "text": " until I actually understood the source code and then I figured everything"}, {"start": 1529.52, "end": 1534.36, "text": " made sense so now hopefully if you go through my code it will be a bit easier"}, {"start": 1534.36, "end": 1540.66, "text": " to understand what happened here just one more quick thing that's that I had"}, {"start": 1540.66, "end": 1548.6000000000001, "text": " to add this basically custom data set because using space tokenizers at the"}, {"start": 1548.6000000000001, "end": 1555.6000000000001, "text": " moment it took me 66 seconds 70 seconds whatever to to re-tokenize the the"}, {"start": 1555.6000000000001, "end": 1560.2, "text": " source and target sentences every single time and so I just decided why not"}, {"start": 1560.2, "end": 1567.5600000000002, "text": " tokenized once save it as a like simple txt file and then instead of"}, {"start": 1567.56, "end": 1571.6799999999998, "text": " re-tokenizing things again just load the txt file and that now takes like two and"}, {"start": 1571.6799999999998, "end": 1577.08, "text": " a half seconds so in a nutshell that's what I did with these data set wrapper"}, {"start": 1577.08, "end": 1583.28, "text": " and this fast translation data set wrapper class so that was a short story"}, {"start": 1583.28, "end": 1589.04, "text": " about like some problems I encountered now like for the end let me just show"}, {"start": 1589.04, "end": 1593.96, "text": " you how the thing functions and how the translation script actually works and I"}, {"start": 1593.96, "end": 1597.52, "text": " think it's really interesting and magical for me because I was usually"}, {"start": 1597.52, "end": 1601.68, "text": " doing computer vision things and then once you see this thing actually working"}, {"start": 1601.68, "end": 1605.24, "text": " and translating from English to German and vice versa it's pretty fascinating"}, {"start": 1605.24, "end": 1609.3400000000001, "text": " if you ask me so I've got up so you basically just have to input the"}, {"start": 1609.3400000000001, "end": 1612.92, "text": " sentence depending so I have an English sentence how are you doing today"}, {"start": 1612.92, "end": 1618.8, "text": " whatever like set as a source sentence and we have the model that was trained"}, {"start": 1618.8, "end": 1623.82, "text": " on IWSLT data set and it's translating from English to German so the name is"}, {"start": 1623.82, "end": 1627.24, "text": " pretty indicative of what the model is doing and where the model was trained on"}, {"start": 1627.24, "end": 1633.76, "text": " so you just have to keep these two in sync with the model so if you change the"}, {"start": 1633.76, "end": 1640.3999999999999, "text": " model to G to E German to English you will just have to set the enum here to"}, {"start": 1640.3999999999999, "end": 1648.28, "text": " G to E instead of E to G that's it once I start this thing let me run it it takes"}, {"start": 1648.28, "end": 1654.6399999999999, "text": " a little bit of time to half seconds to load the the like the the the"}, {"start": 1654.6399999999999, "end": 1660.76, "text": " vocabularies and what cat to load vocabulary and then it'll just translate"}, {"start": 1660.76, "end": 1668.92, "text": " why is it not working okay here it is it took two and a two point six seconds to"}, {"start": 1668.92, "end": 1674.12, "text": " load the data and then you can see the input sentence the source sentence"}, {"start": 1674.12, "end": 1679.32, "text": " tokenized and you see the output sentence we get the scene heute which"}, {"start": 1679.32, "end": 1684.52, "text": " means the same thing pretty much although enum is a German word like a"}, {"start": 1684.52, "end": 1689.56, "text": " polite way to say you so there is nothing stopping me from translating from"}, {"start": 1689.56, "end": 1694.32, "text": " German to English let me just change this thing to German to English let me"}, {"start": 1694.32, "end": 1702.08, "text": " change the model to German to English and now if I input for example we get"}, {"start": 1702.08, "end": 1709.84, "text": " this dia which translates as yeah how are you basically in English let me see"}, {"start": 1709.84, "end": 1715.12, "text": " what the model will say how the model would translate this this time and it"}, {"start": 1715.12, "end": 1719.52, "text": " basically translated this as how are you which is pretty much something called"}, {"start": 1719.52, "end": 1723.9199999999998, "text": " gold translation which means like ground truth translation this looks pretty"}, {"start": 1723.9199999999998, "end": 1728.96, "text": " pretty decent the model is not perfect you can find some sentences which would"}, {"start": 1728.96, "end": 1736.04, "text": " kind of confuse the model so even a simple sentence like a spin I'm"}, {"start": 1736.04, "end": 1741.0, "text": " berlinna which is a sentence like I think Kennedy said when he was back in"}, {"start": 1741.0, "end": 1748.24, "text": " Berlin back in the days and it means I'm a Berliner I'm a I'm person living in"}, {"start": 1748.24, "end": 1756.8, "text": " Berlin and if I input this and run it so the model will actually the thing is the"}, {"start": 1756.8, "end": 1762.68, "text": " model doesn't have a token Berliner inside the vocabulary so that's why it"}, {"start": 1762.68, "end": 1769.0, "text": " will say Berlin instead of say I mean I'm anthropomorphizing the model here so it"}, {"start": 1769.0, "end": 1775.9199999999998, "text": " will output Berlin so I am a Berlin so if the model had Berliner it would"}, {"start": 1775.9199999999998, "end": 1780.52, "text": " output Berliner so that's another another point I want to make here so if I had"}, {"start": 1780.52, "end": 1784.36, "text": " something called byte pairing coding I could the model would be much more"}, {"start": 1784.36, "end": 1789.1599999999999, "text": " expressive in creating these words then at this point of time but that's a"}, {"start": 1789.1599999999999, "end": 1795.9599999999998, "text": " future work and I'll edit hopefully like maybe next week or something so there"}, {"start": 1795.9599999999998, "end": 1799.76, "text": " are many other things I could show you but for the sake of this video not being"}, {"start": 1799.76, "end": 1805.3999999999999, "text": " like one hour long let me just show you how the perf what amount of hardware"}, {"start": 1805.3999999999999, "end": 1810.1599999999999, "text": " requirements do you need in order to run translations not in order to train the"}, {"start": 1810.16, "end": 1814.8400000000001, "text": " model you would need a lot of GPUs to train the model actually had have some"}, {"start": 1814.8400000000001, "end": 1819.4, "text": " comments in the readme please go go ahead and read that part but for like"}, {"start": 1819.4, "end": 1825.16, "text": " just for for translations let me just open the task manager and if I open it"}, {"start": 1825.16, "end": 1832.24, "text": " up if I run the script now we can follow how much GPU is needed to to get the"}, {"start": 1832.24, "end": 1838.28, "text": " translation so it's so it's basically only a small spike so you won't need as"}, {"start": 1838.28, "end": 1843.0, "text": " much hardware like like a really fancy GPU if you're just running translations"}, {"start": 1843.0, "end": 1848.0, "text": " on the other side if you're training this model you'll need a lot of power so"}, {"start": 1848.0, "end": 1853.28, "text": " that that pretty much wraps it up hope you like this video there was a lot of"}, {"start": 1853.28, "end": 1859.28, "text": " information in here so what my next steps will be is I'll start reading"}, {"start": 1859.28, "end": 1863.72, "text": " research papers again I want to explore what state-of-the-art in natural language"}, {"start": 1863.72, "end": 1870.56, "text": " processing now in 2020 so I'll be spending like maybe reading one paper in"}, {"start": 1870.56, "end": 1874.96, "text": " the morning one paper in the evening that amounts to maybe at least ten"}, {"start": 1874.96, "end": 1880.4, "text": " papers a week which is good enough to get me like familiar with the field in"}, {"start": 1880.4, "end": 1886.32, "text": " like 10 days period two weeks so yeah that was pretty much it if you like this"}, {"start": 1886.32, "end": 1891.24, "text": " video go ahead and subscribe to this channel and hit the bell icon to get"}, {"start": 1891.24, "end": 1897.8, "text": " notified when I upload a new video and until next time keep learning"}]