30k subscribers AMA (Ask Me Anything) | Channel Update
❤️ Become The AI Epiphany Patreon ❤️
https://www.patreon.com/theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

Celebrating 30.000 subscribers and doing an ask-me-anything session!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:26 Your feedback matters!
00:52 AMA session kick off
01:08 Question1 - Research engineer vs research scientist?
03:10 Question2 - PhD?
04:36 Question3 - How is living in London?
05:00 Question4 - Most important skills for AI?
06:18 Question5 - Will you be doing podcasts on this channel?
06:50 Question6 - What are you most excited about in ML today?
08:41 Question7 - What do you do to relax?
09:08 Question8 - Replicability crisis, large models?
11:15 Question9 - When will AI finally become conscious?
11:35 Question10 - Experiences so far in DeepMind?
12:26 Question11 - How has my view of the ML changed?
14:41 Question12 - Next big thing?

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#ama #milestone #youtube
 What's up guys, the channel just hit 30,000 subscribers and I'm making this video to answer some of your questions and just to celebrate and reflect over the last period. I'm super thankful for all of you following my channel. I'm learning so much from you as well, just reading the feedback, reading the comments, interacting with you guys is super valuable for me as well, so I'm very grateful for each one of you. Having said that, over the last period I've been focusing more, I guess you noticed, on these ML code walkthrough videos, so do let me know whether you find those interesting. Any comment, feedback, whatnot, what can I improve there, what you want to see more of, whatever pops on your mind, feel free to just kind of comment down below, I'll answer every single comment and yeah, I'm looking forward to your thoughts. So I'm making this video also to just answer some of your questions, so this morning I posted a couple of posts to LinkedIn and Twitter and I collected some of your questions, so I thought just going through them and answering them and hopefully you get something useful out of this video. Okay guys, so let's start with the first one. Ahmet Tek here says, so congratulations, this is a great channel, thanks Ahmet. My question is, should I go for a research engineer position or a research scientist position? What is the difference in roles and who generally has more impact? This is one of the questions where the answer is really, it depends. It depends on the team you're in, it depends on the project you have, it depends on the phase of the project and it depends on your particular role. So there is a lot of overlap between what a research engineer and research scientists do, so again, it varies a lot and sometimes a research engineer will be doing more research than a research scientist in particular phases of the project and vice versa. So it really depends on a particular team you land in. And the second related question from Ahmet as well, also what do you think of being a research engineer for one to three years then becoming a research scientist with a PhD? I feel this is the best thing to do. So I personally encourage you to, so this is just my advice, my personal like thinking on this topic is only if you are passionate about a particular topic and you really want to invest all like four or five plus years investigating this particular research topic then you should go and do the PhD route. Otherwise you probably don't want to do PhD, but that's just my two cents. Touching on the impact topic a little bit, again basically the impact inside of a company is usually measured, so the proxy we use is the the level you're on, so the seniority. And basically I can give you REs who are of higher levels than some RSs and thus they've been more impactful using that that metric. So it really really depends and I wouldn't consider any each like any of these two as a superior to the other one. They're just different and ultimately there is a lot of overlap. So that's those are my two cents. The second question comes from Kevin Crow. The first one is, so if you were doing a machine learning PhD what would be your preferred area of research and why? So obviously I haven't been I haven't done a PhD because that's my personal like choice. I think that it's much better for me to do this self-education path and just have a broader overview of the field rather than zooming into a particular topic. That's just my personal preference but if I were to choose I would probably be well this is kind of not not really ML but related especially nowadays. So somewhere on the intersection like scaling or high performance computing, so HPC, there will be an interest like the compilers or just like how to scale models to well much bigger sizes. That's interesting. Reconciling, learning and causality. That's something that's very interesting and currently people are working on that very hard. So I think I would maybe try that one. Geometric ML is something I'm super passionate about and finally RL. So there is a lot of cool recent projects coming coming from OpenAI and others where like basically we are learning the human preferences using certain RL methods. So I think RL still has a lot to say in the future of AI. The second question from from Kevin is how is living in London? It's super great. I love it. It's such a multicultural city. You can find anything really what you want to do. Like I live in Camden so that area is very nice. I have like Regions Park which is a huge park here in London, fairly nearby and like yeah I'm loving it so far. Thanks for the question Kevin. The next one is from Hightam. So I have one question. What are the skills and knowledge that you think are most important for AI scientists to have in this new era? I'm gonna go a bit meta here and not focus on any particular technology or skill set. So I'm gonna focus on things such as you need to have patience. You need to be willing to learn to constantly learn and self-improve. You need decent communication skills and a lot of I think a lot of engineers and scientists struggle with this one but this is super important. Ultimately you're going to be collaborating with people and you don't want the bottleneck to be your like a lack of communication skills. Some software engineer skills are super recommended no matter the role to be honest. Even if you're RE, RS or well if you're a software engineer obviously you need software engineer skills but like that's maybe like a basic skill, a fundamental skill and of course some domain expertise depending on the role you're working in. There'll be different things you'll want to learn depending whether you're doing computer vision or RL or this or that you'll want to learn different frameworks, different technologies etc etc. Okay so the next one is from Marco. Have you thought about doing interviews on your channel and by interviews I assume Marco means podcasts here. I'm planning to do those maybe some sometime in the future or I might occasionally surprise you with some interesting guests on this channel but so far I don't have any any clear plans on that. I will have to skip the the second one here because I do work a deep mind so yeah. So the next one comes from Igor. What about are you most excited in deep learning today and I have to say so again my personal preference here is just scaling various existing approaches and the reasoning behind this and I don't want to sound like that like AGI is coming guy but my reasoning here is we haven't even scaled to the scale of a human brain. So human brain has roughly let's say 80 billion neurons and on average there is maybe 10,000 connections so all in all there is like what one quadrillion like synapses in the brain and we are nowhere close to that to that scale and since we are still not seeing any situation why would we stop plus the brain size is a fictional limit why not go 10x the brain size or 100x the brain size as long as we don't see the situation I think it's a viable path forward not the only one but like a one that looks to be very promising and so I think we we need to to kind of build up those muscles so I like to think about that similarly to how gaming like inadvertently was the cause for us to build the GPU muscles and then we could use the GPU muscles to do deep learning and similarly here if we just develop this skill of building huge models maybe later on when when some novel research idea comes like pops into the well into the consciousness of the of the research community then we can easily scale it up and ultimately we know that all of the biological intelligence that exists on this world does have bunch of neurons and bunch of bunch of connections so we knew we know that scale is a necessary factor probably not sufficient but but yeah there was a bit longer answer there the next one what do you do after work in London before you go to make AI videos how do you relax so for me it's usually doing power naps working out I started running a lot here in London I also watch podcasts with my girlfriend and every now and then I try and travel around so I recently like literally a week ago came back from from my Italy trip so that was very fairly refreshing just kind of charging the batteries okay let's go for the next one in a lot of papers there is a replicability issue as the models to get bigger and bigger and significantly more powerful hardware is required if you were to organize conferences would you try to do something regarding that do you believe that good papers that required insane amount of computational power would have still been possible if they weren't to have access to the required hardware for example the lead to or GPT 3 so there is multiple questions bundled into this one I'm gonna try and dissect it may be starting from the from the last one so it's obvious that these models would not have been possible without the scale because we do know that certain like skills emerge with the scale and has all of the all of this scale is all you need hype going on I think that like a big science like effort is a good example of what like an open source what a community could do when we come together and and build something together so if you're not familiar with this there was this huge language model 176 billion parameters called bloom that came from big science and that's an example of how people coming together can build awesome stuff you lose the AI is also a good example where we're just passionate individuals applying for I think it was the TPU research crawler something you can apply for that and get some free TPUs and then literally they meet they built a model that had 20 billion parameters the GPT Neo X 20b so I do think this is a problem especially when we when we realize that skills do emerge with scale so here are some I gave you some examples like you lose very I am big science effort as examples of what we can do but there is also a lot of things we can do a lot of research directions that don't need that type of a scale but like I don't want to be that guy saying so I don't want to be that guy downplaying the importance of having like access to a nice amount of resources so yeah these are just some ideas of how you can maybe join those organizations or yeah hopefully that answers your question the one from Stefan here when will AI finally become conscious asking for a friend's grandmother who is very concerned so I tell the grandma not to be like worried it won't be conscious anytime soon but slightly conscious who knows okay so Brian Pulfer here says congratulations Alexa I'd love to know more about your experience in a top AI company like DeepMind and how it has affected your view about ML if at all I'd also be curious to know your opinion about the next big thing in the field as a DeepMind employee you probably have a sense for the field that others don't congrats again and keep up the amazing work you're doing thanks Brian so so my experience so far with DeepMind is that the company really cares about the employees that the environment is amazing the people are amazing I mean you do have a bunch of world-class exports around you and just learning by osmosis and reaching out to those to those folks is super valuable you can you can they're usually super responsible like I literally had a conversation with I am good fellow a couple actually yesterday so so yeah I really love it so far okay on to the second part of your question basically how has it affected my view on the ML field I think just being present on the on the edge of what's going on in the in the research in our community I realized like how crazy we have this commonality of engineers experience when when I say that I mean suffering if you take a look at some of the of the recent Chronicles that for example let me show you let me open up a page here on github so here is the like Chronicles from the like metas training of this huge LLM that this was like hundred seventy five billion parameter big you can see it like problems that they are facing like like cluster deletion let me read this for you so given the holidays we requested a pool of 12 buffer nodes to guard against hardware failures two machines go down every day in the process of replenishing this pool the cloud providers support team accidentally deleted our entire cluster on December 21st so things like this happen all too often even in companies such as meta and and others let me show you another example from from the big science effort I mentioned the blue model so these guys literally had some problems with the stability you can see the the spikes in the loss so the the loss would go up and never return back to its initial value neither go down obviously so the training would diverge and one of the things they noticed analyzing the spikes is that they literally have so they said here two million backslash only samples in our data set so literally like a string of two million backslashes that that somehow nobody noticed up until this point when the when the when the training started diverging and so problems like this pop up all the time even if you work on in the best companies in the world wherever you are all people like all engineers and scientists are struggling and having to deal with various different issues and I think having that getting that perspective not just internally in deep mind but this is I showed you here some external examples you can see how how the everyday work of an engineer or a scientist looks completely different compared to what people usually think as for the next big thing I can only give my personal opinion here obviously I'm not representing deep mind here on this channel I as I said I'm a believer in in scaling the current existing approaches not saying those are sufficient I'm just saying those are necessary and like building those muscles is something that's gonna be super useful over the long run that's it guys those are the questions I kind of selected thanks a lot for asking those questions again leave the feedback down below if you have any ideas any suggestions what I should add to my channel or improve or change or whatnot feel free to just comment down below and yeah having said that subscribe to this channel if you haven't let's go to 50k let's go to 100 K and beyond so until next time bye bye
[{"start": 0.0, "end": 6.12, "text": " What's up guys, the channel just hit 30,000 subscribers and I'm making this"}, {"start": 6.12, "end": 10.56, "text": " video to answer some of your questions and just to celebrate and reflect over"}, {"start": 10.56, "end": 16.04, "text": " the last period. I'm super thankful for all of you following my channel. I'm"}, {"start": 16.04, "end": 19.88, "text": " learning so much from you as well, just reading the feedback, reading the"}, {"start": 19.88, "end": 24.2, "text": " comments, interacting with you guys is super valuable for me as well, so I'm"}, {"start": 24.2, "end": 28.2, "text": " very grateful for each one of you. Having said that, over the last period"}, {"start": 28.2, "end": 33.24, "text": " I've been focusing more, I guess you noticed, on these ML code walkthrough"}, {"start": 33.24, "end": 38.879999999999995, "text": " videos, so do let me know whether you find those interesting. Any comment,"}, {"start": 38.879999999999995, "end": 43.24, "text": " feedback, whatnot, what can I improve there, what you want to see more of,"}, {"start": 43.24, "end": 46.96, "text": " whatever pops on your mind, feel free to just kind of comment down below, I'll"}, {"start": 46.96, "end": 51.519999999999996, "text": " answer every single comment and yeah, I'm looking forward to your thoughts."}, {"start": 51.519999999999996, "end": 55.519999999999996, "text": " So I'm making this video also to just answer some of your questions, so this"}, {"start": 55.52, "end": 60.64, "text": " morning I posted a couple of posts to LinkedIn and Twitter and I collected"}, {"start": 60.64, "end": 64.60000000000001, "text": " some of your questions, so I thought just going through them and answering them"}, {"start": 64.60000000000001, "end": 68.4, "text": " and hopefully you get something useful out of this video. Okay guys, so let's"}, {"start": 68.4, "end": 74.08, "text": " start with the first one. Ahmet Tek here says, so congratulations, this is a great"}, {"start": 74.08, "end": 78.56, "text": " channel, thanks Ahmet. My question is, should I go for a research engineer"}, {"start": 78.56, "end": 82.48, "text": " position or a research scientist position? What is the difference in roles"}, {"start": 82.48, "end": 88.0, "text": " and who generally has more impact? This is one of the questions where the"}, {"start": 88.0, "end": 92.68, "text": " answer is really, it depends. It depends on the team you're in, it depends on the"}, {"start": 92.68, "end": 96.92, "text": " project you have, it depends on the phase of the project and it depends on your"}, {"start": 96.92, "end": 100.68, "text": " particular role. So there is a lot of overlap between what a research"}, {"start": 100.68, "end": 106.04, "text": " engineer and research scientists do, so again, it varies a lot and sometimes a"}, {"start": 106.04, "end": 109.96000000000001, "text": " research engineer will be doing more research than a research scientist in"}, {"start": 109.96, "end": 114.44, "text": " particular phases of the project and vice versa. So it really depends on a"}, {"start": 114.44, "end": 119.32, "text": " particular team you land in. And the second related question from Ahmet as"}, {"start": 119.32, "end": 124.08, "text": " well, also what do you think of being a research engineer for one to three years"}, {"start": 124.08, "end": 128.44, "text": " then becoming a research scientist with a PhD? I feel this is the best thing to"}, {"start": 128.44, "end": 135.24, "text": " do. So I personally encourage you to, so this is just my advice, my personal like"}, {"start": 135.24, "end": 139.92, "text": " thinking on this topic is only if you are passionate about a particular topic"}, {"start": 139.92, "end": 146.07999999999998, "text": " and you really want to invest all like four or five plus years investigating"}, {"start": 146.07999999999998, "end": 151.32, "text": " this particular research topic then you should go and do the PhD route. Otherwise"}, {"start": 151.32, "end": 157.67999999999998, "text": " you probably don't want to do PhD, but that's just my two cents. Touching on the"}, {"start": 157.67999999999998, "end": 162.95999999999998, "text": " impact topic a little bit, again basically the impact inside of a company"}, {"start": 162.95999999999998, "end": 167.27999999999997, "text": " is usually measured, so the proxy we use is the the level you're on, so the"}, {"start": 167.28, "end": 172.0, "text": " seniority. And basically I can give you REs who are of higher levels than"}, {"start": 172.0, "end": 177.28, "text": " some RSs and thus they've been more impactful using that that metric. So it"}, {"start": 177.28, "end": 182.84, "text": " really really depends and I wouldn't consider any each like any of these two"}, {"start": 182.84, "end": 187.56, "text": " as a superior to the other one. They're just different and ultimately there is a"}, {"start": 187.56, "end": 191.88, "text": " lot of overlap. So that's those are my two cents. The second question comes from"}, {"start": 191.88, "end": 198.6, "text": " Kevin Crow. The first one is, so if you were doing a machine learning PhD what"}, {"start": 198.6, "end": 204.16, "text": " would be your preferred area of research and why? So obviously I haven't been I"}, {"start": 204.16, "end": 209.88, "text": " haven't done a PhD because that's my personal like choice. I think that it's"}, {"start": 209.88, "end": 214.12, "text": " much better for me to do this self-education path and just have a"}, {"start": 214.12, "end": 217.72, "text": " broader overview of the field rather than zooming into a particular topic."}, {"start": 217.72, "end": 223.12, "text": " That's just my personal preference but if I were to choose I would probably be"}, {"start": 223.12, "end": 229.36, "text": " well this is kind of not not really ML but related especially nowadays. So"}, {"start": 229.36, "end": 234.12, "text": " somewhere on the intersection like scaling or high performance computing, so"}, {"start": 234.12, "end": 238.8, "text": " HPC, there will be an interest like the compilers or just like how to scale"}, {"start": 238.8, "end": 244.64, "text": " models to well much bigger sizes. That's interesting. Reconciling, learning and"}, {"start": 244.64, "end": 248.67999999999998, "text": " causality. That's something that's very interesting and currently people are"}, {"start": 248.67999999999998, "end": 254.88, "text": " working on that very hard. So I think I would maybe try that one. Geometric ML is"}, {"start": 254.88, "end": 259.12, "text": " something I'm super passionate about and finally RL. So there is a lot of cool"}, {"start": 259.12, "end": 264.64, "text": " recent projects coming coming from OpenAI and others where like basically"}, {"start": 264.64, "end": 270.0, "text": " we are learning the human preferences using certain RL methods. So I think RL"}, {"start": 270.0, "end": 275.72, "text": " still has a lot to say in the future of AI. The second question from from Kevin"}, {"start": 275.72, "end": 280.52, "text": " is how is living in London? It's super great. I love it. It's such a multicultural"}, {"start": 280.52, "end": 286.8, "text": " city. You can find anything really what you want to do. Like I live in"}, {"start": 286.8, "end": 292.64, "text": " Camden so that area is very nice. I have like Regions Park which is a huge park"}, {"start": 292.64, "end": 299.08, "text": " here in London, fairly nearby and like yeah I'm loving it so far. Thanks for the"}, {"start": 299.08, "end": 305.28, "text": " question Kevin. The next one is from Hightam. So I have one"}, {"start": 305.28, "end": 308.15999999999997, "text": " question. What are the skills and knowledge that you think are most"}, {"start": 308.15999999999997, "end": 314.4, "text": " important for AI scientists to have in this new era? I'm gonna go a bit meta"}, {"start": 314.4, "end": 319.32, "text": " here and not focus on any particular technology or skill set. So I'm gonna"}, {"start": 319.32, "end": 325.28, "text": " focus on things such as you need to have patience. You need to be willing to learn"}, {"start": 325.28, "end": 331.4, "text": " to constantly learn and self-improve. You need decent communication skills and a"}, {"start": 331.4, "end": 334.64, "text": " lot of I think a lot of engineers and scientists struggle with this one but"}, {"start": 334.64, "end": 337.91999999999996, "text": " this is super important. Ultimately you're going to be collaborating with"}, {"start": 337.91999999999996, "end": 342.0, "text": " people and you don't want the bottleneck to be your like a"}, {"start": 342.0, "end": 347.03999999999996, "text": " lack of communication skills. Some software engineer skills are super"}, {"start": 347.03999999999996, "end": 351.44, "text": " recommended no matter the role to be honest. Even if you're RE, RS or"}, {"start": 351.44, "end": 354.08, "text": " well if you're a software engineer obviously you need software"}, {"start": 354.08, "end": 359.59999999999997, "text": " engineer skills but like that's maybe like a basic skill, a fundamental skill"}, {"start": 359.59999999999997, "end": 365.76, "text": " and of course some domain expertise depending on the role you're"}, {"start": 365.76, "end": 369.59999999999997, "text": " working in. There'll be different things you'll want to learn depending whether"}, {"start": 369.59999999999997, "end": 374.0, "text": " you're doing computer vision or RL or this or that you'll want to learn"}, {"start": 374.0, "end": 379.52, "text": " different frameworks, different technologies etc etc. Okay so the next"}, {"start": 379.52, "end": 385.59999999999997, "text": " one is from Marco. Have you thought about doing interviews on your channel and by"}, {"start": 385.59999999999997, "end": 391.12, "text": " interviews I assume Marco means podcasts here. I'm planning to do those maybe"}, {"start": 391.12, "end": 396.4, "text": " some sometime in the future or I might occasionally surprise you with some"}, {"start": 396.4, "end": 401.52, "text": " interesting guests on this channel but so far I don't have any any clear plans on"}, {"start": 401.52, "end": 406.47999999999996, "text": " that. I will have to skip the the second one here because I do work a deep mind"}, {"start": 406.48, "end": 415.04, "text": " so yeah. So the next one comes from Igor. What about are you most"}, {"start": 415.04, "end": 420.96000000000004, "text": " excited in deep learning today and I have to say so again my personal"}, {"start": 420.96000000000004, "end": 425.36, "text": " preference here is just scaling various existing approaches and the reasoning"}, {"start": 425.36, "end": 430.8, "text": " behind this and I don't want to sound like that like AGI is coming guy but"}, {"start": 430.8, "end": 435.76, "text": " my reasoning here is we haven't even scaled to the scale of a human"}, {"start": 435.76, "end": 440.56, "text": " brain. So human brain has roughly let's say 80 billion neurons and on average"}, {"start": 440.56, "end": 444.32, "text": " there is maybe 10,000 connections so all in all there is like what one"}, {"start": 444.32, "end": 450.56, "text": " quadrillion like synapses in the brain and we are nowhere close to that to that"}, {"start": 450.56, "end": 456.4, "text": " scale and since we are still not seeing any situation why would we stop plus the"}, {"start": 456.4, "end": 461.44, "text": " brain size is a fictional limit why not go 10x the brain size or 100x the brain"}, {"start": 461.44, "end": 466.64, "text": " size as long as we don't see the situation I think it's a viable path"}, {"start": 466.64, "end": 472.16, "text": " forward not the only one but like a one that looks to be very promising and so I"}, {"start": 472.16, "end": 476.4, "text": " think we we need to to kind of build up those muscles so I like to think about"}, {"start": 476.4, "end": 483.12, "text": " that similarly to how gaming like inadvertently was the cause for us to"}, {"start": 483.12, "end": 487.4, "text": " build the GPU muscles and then we could use the GPU muscles to do deep learning"}, {"start": 487.4, "end": 492.91999999999996, "text": " and similarly here if we just develop this skill of building huge models maybe"}, {"start": 492.91999999999996, "end": 498.79999999999995, "text": " later on when when some novel research idea comes like pops into the well into"}, {"start": 498.79999999999995, "end": 503.71999999999997, "text": " the consciousness of the of the research community then we can easily scale it up"}, {"start": 503.71999999999997, "end": 509.32, "text": " and ultimately we know that all of the biological intelligence that exists on"}, {"start": 509.32, "end": 513.1999999999999, "text": " this world does have bunch of neurons and bunch of bunch of connections so we"}, {"start": 513.1999999999999, "end": 517.36, "text": " knew we know that scale is a necessary factor probably not sufficient but"}, {"start": 517.36, "end": 521.28, "text": " but yeah there was a bit longer answer there the next one what do you do after"}, {"start": 521.28, "end": 526.36, "text": " work in London before you go to make AI videos how do you relax so for me it's"}, {"start": 526.36, "end": 532.52, "text": " usually doing power naps working out I started running a lot here in London I"}, {"start": 532.52, "end": 538.72, "text": " also watch podcasts with my girlfriend and every now and then I try and travel"}, {"start": 538.72, "end": 543.04, "text": " around so I recently like literally a week ago came back from from my Italy"}, {"start": 543.04, "end": 548.04, "text": " trip so that was very fairly refreshing just kind of charging the batteries okay"}, {"start": 548.04, "end": 552.0999999999999, "text": " let's go for the next one in a lot of papers there is a replicability issue as"}, {"start": 552.0999999999999, "end": 555.88, "text": " the models to get bigger and bigger and significantly more powerful hardware is"}, {"start": 555.88, "end": 559.88, "text": " required if you were to organize conferences would you try to do"}, {"start": 559.88, "end": 563.76, "text": " something regarding that do you believe that good papers that required insane"}, {"start": 563.76, "end": 568.3199999999999, "text": " amount of computational power would have still been possible if they weren't to"}, {"start": 568.32, "end": 573.08, "text": " have access to the required hardware for example the lead to or GPT 3 so there is"}, {"start": 573.08, "end": 577.0400000000001, "text": " multiple questions bundled into this one I'm gonna try and dissect it may be"}, {"start": 577.0400000000001, "end": 582.32, "text": " starting from the from the last one so it's obvious that these models would not"}, {"start": 582.32, "end": 586.9200000000001, "text": " have been possible without the scale because we do know that certain like"}, {"start": 586.9200000000001, "end": 591.12, "text": " skills emerge with the scale and has all of the all of this scale is all you need"}, {"start": 591.12, "end": 599.88, "text": " hype going on I think that like a big science like effort is a good"}, {"start": 599.88, "end": 604.76, "text": " example of what like an open source what a community could do when we come"}, {"start": 604.76, "end": 607.68, "text": " together and and build something together so if you're not familiar with"}, {"start": 607.68, "end": 613.64, "text": " this there was this huge language model 176 billion parameters called bloom that"}, {"start": 613.64, "end": 618.12, "text": " came from big science and that's an example of how people coming together"}, {"start": 618.12, "end": 622.72, "text": " can build awesome stuff you lose the AI is also a good example where we're just"}, {"start": 622.72, "end": 627.28, "text": " passionate individuals applying for I think it was the TPU research crawler"}, {"start": 627.28, "end": 632.0, "text": " something you can apply for that and get some free TPUs and then literally they"}, {"start": 632.0, "end": 637.24, "text": " meet they built a model that had 20 billion parameters the GPT Neo X 20b so"}, {"start": 637.24, "end": 642.88, "text": " I do think this is a problem especially when we when we realize that skills do"}, {"start": 642.88, "end": 647.32, "text": " emerge with scale so here are some I gave you some examples like you lose"}, {"start": 647.32, "end": 652.2, "text": " very I am big science effort as examples of what we can do but there is also a"}, {"start": 652.2, "end": 655.72, "text": " lot of things we can do a lot of research directions that don't need that"}, {"start": 655.72, "end": 660.1600000000001, "text": " type of a scale but like I don't want to be that guy saying so I don't want to be"}, {"start": 660.1600000000001, "end": 666.72, "text": " that guy downplaying the importance of having like access to a nice amount of"}, {"start": 666.72, "end": 671.6800000000001, "text": " resources so yeah these are just some ideas of how you can maybe join those"}, {"start": 671.6800000000001, "end": 676.96, "text": " organizations or yeah hopefully that answers your question the one from"}, {"start": 676.96, "end": 681.6800000000001, "text": " Stefan here when will AI finally become conscious asking for a friend's"}, {"start": 681.6800000000001, "end": 687.76, "text": " grandmother who is very concerned so I tell the grandma not to be like worried"}, {"start": 687.76, "end": 694.36, "text": " it won't be conscious anytime soon but slightly conscious who knows okay so"}, {"start": 694.36, "end": 700.6, "text": " Brian Pulfer here says congratulations Alexa I'd love to know more about your"}, {"start": 700.6, "end": 704.48, "text": " experience in a top AI company like DeepMind and how it has affected your"}, {"start": 704.48, "end": 709.32, "text": " view about ML if at all I'd also be curious to know your opinion about the"}, {"start": 709.32, "end": 713.08, "text": " next big thing in the field as a DeepMind employee you probably have a"}, {"start": 713.08, "end": 716.8000000000001, "text": " sense for the field that others don't congrats again and keep up the amazing"}, {"start": 716.8000000000001, "end": 722.36, "text": " work you're doing thanks Brian so so my experience so far with DeepMind is that"}, {"start": 722.36, "end": 725.84, "text": " the company really cares about the employees that the environment is"}, {"start": 725.84, "end": 730.0, "text": " amazing the people are amazing I mean you do have a bunch of world-class"}, {"start": 730.0, "end": 734.6, "text": " exports around you and just learning by osmosis and reaching out to those to"}, {"start": 734.6, "end": 738.56, "text": " those folks is super valuable you can you can they're usually super"}, {"start": 738.56, "end": 741.96, "text": " responsible like I literally had a conversation with I am good fellow a"}, {"start": 741.96, "end": 747.4, "text": " couple actually yesterday so so yeah I really love it so far okay on to the"}, {"start": 747.4, "end": 753.04, "text": " second part of your question basically how has it affected my view on the ML"}, {"start": 753.04, "end": 758.6, "text": " field I think just being present on the on the edge of what's going on in the in"}, {"start": 758.6, "end": 765.48, "text": " the research in our community I realized like how crazy we have this commonality"}, {"start": 765.48, "end": 771.72, "text": " of engineers experience when when I say that I mean suffering if you take a look"}, {"start": 771.72, "end": 775.8000000000001, "text": " at some of the of the recent Chronicles that for example let me show you let me"}, {"start": 775.8000000000001, "end": 782.44, "text": " open up a page here on github so here is the like Chronicles from the like"}, {"start": 782.44, "end": 786.86, "text": " metas training of this huge LLM that this was like hundred seventy five"}, {"start": 786.86, "end": 790.4, "text": " billion parameter big you can see it like problems that they are facing like"}, {"start": 790.4, "end": 794.0, "text": " like cluster deletion let me read this for you so given the holidays we"}, {"start": 794.0, "end": 798.14, "text": " requested a pool of 12 buffer nodes to guard against hardware failures two"}, {"start": 798.14, "end": 801.98, "text": " machines go down every day in the process of replenishing this pool the"}, {"start": 801.98, "end": 806.16, "text": " cloud providers support team accidentally deleted our entire cluster"}, {"start": 806.16, "end": 811.52, "text": " on December 21st so things like this happen all too often even in companies"}, {"start": 811.52, "end": 815.96, "text": " such as meta and and others let me show you another example from from the big"}, {"start": 815.96, "end": 820.58, "text": " science effort I mentioned the blue model so these guys literally had some"}, {"start": 820.58, "end": 824.76, "text": " problems with the stability you can see the the spikes in the loss so the the"}, {"start": 824.76, "end": 829.52, "text": " loss would go up and never return back to its initial value neither go down"}, {"start": 829.52, "end": 833.74, "text": " obviously so the training would diverge and one of the things they noticed"}, {"start": 833.74, "end": 837.08, "text": " analyzing the spikes is that they literally have so they said here two"}, {"start": 837.08, "end": 841.64, "text": " million backslash only samples in our data set so literally like a string of"}, {"start": 841.64, "end": 847.9399999999999, "text": " two million backslashes that that somehow nobody noticed up until this"}, {"start": 847.9399999999999, "end": 851.14, "text": " point when the when the when the training started diverging and so"}, {"start": 851.14, "end": 855.96, "text": " problems like this pop up all the time even if you work on in the best"}, {"start": 855.96, "end": 860.8, "text": " companies in the world wherever you are all people like all engineers and"}, {"start": 860.8, "end": 864.96, "text": " scientists are struggling and having to deal with various different issues and I"}, {"start": 864.96, "end": 869.68, "text": " think having that getting that perspective not just internally in deep"}, {"start": 869.68, "end": 874.7199999999999, "text": " mind but this is I showed you here some external examples you can see how how"}, {"start": 874.7199999999999, "end": 879.0799999999999, "text": " the everyday work of an engineer or a scientist looks completely different"}, {"start": 879.0799999999999, "end": 883.9599999999999, "text": " compared to what people usually think as for the next big thing I can only give"}, {"start": 883.9599999999999, "end": 887.7199999999999, "text": " my personal opinion here obviously I'm not representing deep mind here on this"}, {"start": 887.7199999999999, "end": 895.3199999999999, "text": " channel I as I said I'm a believer in in scaling the current existing approaches"}, {"start": 895.32, "end": 900.72, "text": " not saying those are sufficient I'm just saying those are necessary and like"}, {"start": 900.72, "end": 904.8000000000001, "text": " building those muscles is something that's gonna be super useful over the"}, {"start": 904.8000000000001, "end": 909.84, "text": " long run that's it guys those are the questions I kind of selected thanks a"}, {"start": 909.84, "end": 914.32, "text": " lot for asking those questions again leave the feedback down below if you"}, {"start": 914.32, "end": 918.6800000000001, "text": " have any ideas any suggestions what I should add to my channel or improve or"}, {"start": 918.6800000000001, "end": 923.8000000000001, "text": " change or whatnot feel free to just comment down below and yeah having said"}, {"start": 923.8, "end": 928.12, "text": " that subscribe to this channel if you haven't let's go to 50k let's go to 100"}, {"start": 928.12, "end": 957.24, "text": " K and beyond so until next time bye bye"}]