Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video I walk you through what the protein folding problem is, I dig deep into the AlphaFold1 paper, and finally, I take a glimpse and I predict what AlphaFold2 may look like.

This thing is huge. Hopefully, it will accelerate the pace of solving the COVID-19 pandemic as well.

You'll learn about:
✔️ Recap of essential biology
✔️ What is the protein folding problem?
✔️ How does AlphaFold1 exactly work
✔️ How does AlphaFold2 probably work

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ DeepMind on AlphaFold2: https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology
✅ DeepMind on COVID-19: https://deepmind.com/research/open-source/computational-predictions-of-protein-structures-associated-with-COVID-19

2 more useful DeepMind blogs:
✅ https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery
✅ https://deepmind.com/research/case-studies/alphafold

✅ What is protein? https://www.youtube.com/watch?v=wvTv8TqWC48&ab_channel=RCSBProteinDataBank
✅ How are proteins created? https://www.youtube.com/watch?v=gG7uCskUOrA&ab_channel=yourgenome

✅ Nature's blog on AlphaFold2: https://www.nature.com/articles/d41586-020-03348-4
✅ Christian Anfinsen's Nobel prize lecture: https://www.nobelprize.org/uploads/2018/06/anfinsen-lecture.pdf

✅ Charts for sequenced proteins: https://www.ebi.ac.uk/uniprot/TrEMBLstats
✅ PDB sequenced vs 3D structure found: https://www.rcsb.org/stats/growth/growth-released-structures

✅ AlphaFold1 paper: https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D
...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Enter the AlphaFold2
03:40 Nobel prize winner Christian Anfinson, biology recap
05:50 Experimental methods (x-ray crystallography, etc.) and some stats
08:24 Why do we care? (Curing diseases, new materials, etc.) 
10:10 A deep dive into the AlphaFold2 paper
13:30 Overview of the deep learning pipeline
17:30 Understanding torsion angles and beta carbon
21:00 How do distance maps (distograms) work?
26:10 Geometric model and optimization (L-BFGS)
29:10 Primary, secondary, tertiary structures
31:30 Initialization procedure
32:50 Overview of the whole pipeline
33:54 AlphaFold2 predictions (paper still not published)
37:10 COVID pandemic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#alphafold2 #deepmind #biology
 It will change everything. DeepMind's AI makes a gigantic leap in solving protein structures. So, I guess all of you heard about this so far, or most of you. And basically what happened is that DeepMind released, published their results on this year's protein folding challenge. And their method alpha fold, actually the second iteration of their algorithm alpha fold, kind of ruined the whole, like, beat everybody and got the best scores. And we can see, so this is the blog that they published so far, they still haven't published the paper. The paper will come probably in maybe a year or something, because the last paper for the alpha fold one came out in December 2019, where the competition happened in 2018. So, we'll have to wait some time to get to that, to get the paper. And I basically can see the scores throughout the years, so that's a biennial competition called CASP. And it happens, so it happens every two years. And you can see here that in 2018 their method alpha fold one also dominated the competition, but this year they crossed 90 on this global distance test metric, which pretty much tells you how good the alignment between the experimentally found protein structure and the computationally found protein structures are. And I'll first, so the idea of this video will be to first explain you what the problem is, so what the protein folding problem is, give you a bit about history and background, and how proteins work and everything. And then my idea is to go into the alpha fold one paper, because that's what we have so far, and I'll explain how it works, how the deep learning method works, so there'll be a deep, in-depth explanation of the paper. And I'll give some assumptions of what will happen with the alpha fold two, what the method is, they only have one chart, so I'll try and deduce something out of it. So basically, so why this is interesting is that 90 is considered as something that's like, if you get to 90 or above, you're pretty much comparable to experimental methods, the reason being they are also not perfect, and they do have their measuring uncertainties. So here we can see, let me zoom in a bit, here we can see what the problem is. Basically, the green thing is the pseudo ground truth, so that means somebody found that one using some of the experimental methods, and I'll tell you a bit more about those. And the blue one, so the blue one is a computational prediction that DeepMind's alpha fold two made, and you can see it's pretty, pretty close. The results are really, really good, and you can see 90.7, 93.3, and these are just the names of the proteins T1049, whatever. So let me kind of step back a little bit here and tell you something about how the problem came to be. So we've been trying to solve this problem in biology for 50 years already, and so this is kind of a really big, big thing that happened, so yeah, the hype is real this time. First of all, there's this guy called Christian Anfinsen, and you can see in 1972 he gave this Nobel Prize lecture. So he basically got his Nobel Prize because he showed that the function of the protein is pretty much uniquely determined by its 3D structure, which is really interesting. So here is a really cool example of hemoglobin. The hemoglobin forms a pocket to hold heme, a small molecule with an iron atom in the center that binds to oxygen. So basically, like, the structure is made so that this protein, the hemoglobin, has a small pocket where this molecule that has, like, an iron atom which binds to, like, the oxygen and thus transports oxygen throughout our body, like, just that molecule fits so nicely in this hemoglobin. So that's a really plastic example of how the structure determines the function. That's a pretty obvious one. So there's this cool example of this molecule that has this Y shape, and the molecule attaches to bacteria or viruses and thus stacks those cells for destruction in the immune system. So let me give you just a brief example of how the proteins are created. So I like to think of this as, like, a kitchen where the ribosome is basically this macromolecule acts as a cook. We have the messenger RNA, which is pretty much the receipt, and then we have amino acids, which are the ingredients, which are transported by this thing called the transport RNA. So basically, the messenger RNA contains this linear sequence of amino acids, and it gets translated so the receipt gets read by this ribosome, and we form, like, a chain of proteins which slowly start folding because different parts, different amino acids have different charges, and so they attract or repel, and thus we get that 3D structure. So that's basically how the protein is formed, and now we're trying to figure out this 3D structure computationally. Now, the thing is, how do people find the 3D structures now that we know that it's really important for, like, for figuring out the function of the proteins? How do we find the structure of the protein? So there are three methods that are used extensively in different labs, and those are mainly X-ray diffraction, nuclear magnetic resonance, and this electron microscopy. And as you can see, the X-ray diffraction was so far had the highest amount of proteins were, like, protein structures were determined using this method. Although it does have its own flaws, it cannot solve, so depending on the size of the protein, it has its constraints, so it can't solve everything. Now, these two are getting more popular lately, but they are still super expensive and really slow methods to obtain the 3D structure. So that's why the computational biology and these computational methods for figuring out the structure were so important. So if we take a look at this chart here, we can see that the number of proteins we sequenced, successfully sequenced, is around 200 million now in 2020. So it's been rising, like, so those are the linear sequences that are encoded in the DNA, so, but we still don't have the 3D structure for all of those. And if we take a look at this chart, we can see there is this thing called a protein database, which is really important because that's the data set that's been used in the AlphaFold paper, and we'll get to it in a couple of minutes. But basically here, you can see that, like, we have, so we have a lot of sequences being kind of ingested into this data set, but the number of structures is actually a lot smaller. So here, 2020, we can see only 12,000 structures are there, but we have, like, much more sequences. So that was the motivation for why we want to accelerate the pace of finding 3D structures. The experimental one is slow, it's really expensive, so, yeah, we need an alternative. So, again, why we care about figuring out the structure, the 3D structure of the protein is because, as you can see on DeepMind's blog, an error in the genetic recipe may result in a malformed protein which could result in disease or death for an organism. Many diseases, therefore, are fundamentally linked to proteins. So diseases like diabetes, like dementia, Parkinson's, Alzheimer's, cancer, some of the diseases that are known to be, like, plaguing the humankind for decades or more can probably be solved by just us knowing the 3D structures. So that's why it's super important to, so that's why this problem is super important. Aside from that, also, we could develop much better materials if we knew how to construct and understand the shapes of different proteins, and we could create, like, plastic-eating enzymes and thus reduce the pollution, like, on the global level, which is something we do care about, obviously. So that was the background story. It was a bit longer. I assume most of my audiences, like, machine learning, have machine learning background, so I felt I need to really give you a nice overview of what the problem is and everything that goes inside, and, like, give you some numbers and methods, etc. So let's start exploring the Alpha Fold 1 now. As you can see here, this is an animation that DeepMind provided, which basically shows how, during their optimization procedure, which we'll get to in a sec, like, folds the proteins, so it starts from a semi-unfolded protein strand, which then slowly takes its 3D shape in space. Okay, let's overview the Alpha Fold 1 paper. You can see, so, the title is Alpha Fold Improved Protein Structure Prediction Using Potentials from Deep Learning. They had, like, a bunch of folks working on this project, as it's pretty common for huge projects like this one. So, they started in 2016, so it's already been, like, a four-year effort of many smart people. So, I've already mentioned it's DeepMind. They're based in London, and, okay. So, the paper is pretty complicated. There's a lot of text and a lot of domain knowledge, so I'll just try and extract the Deep Learning side of it, and hopefully that will give you, like, a good understanding of how the method works. Let's start with this chart. So, basically, on this chart here, they just show that they are better than other groups. So, many other research groups were participating in this in 2018, in Cusp13 Challenge, that was the name of the challenge. And, basically, the green lines represent the groups, those other groups, and the blue line is DeepMind. And so, on the, basically, on the x-axis, we have something called TM score. So, there are a bunch of metrics for determining how good the structure prediction is. So, one is TM. There are other scores, like we saw the Global Distance Test score. There are some, like, IDDT or something. Yeah, there are a lot of different scores, and basically, so, FM Domain Count, Domain is some, like, fancy name for protein, pretty much. So, here, in this free modeling domain, we have 43 sequences which we needed to figure the structure for. And you can see that, on average, the DeepMind had a lot higher TM score than others. So, that's what this thing is. And now, I should probably tell you what the difference between free modeling and template modeling is. So, template modeling is when you're trying to predict the structure for a sequence that has similar sequences whose structure we already know. Now, the thing is, once you know some similar sequence, you can kind of assume that the structure of your sequence will be similar to that one. So, you just need to find where the sequence differs and tweak, kind of tweak the structure, so, starting from that, maybe that structure. So, that's a template modeling. So, I'm simplifying a little bit here, but, like, that's the rough notion. Again, here, on this chart, they just displayed five different proteins, and they, again, show that, like, the blue dots that DeepMind is crushing it. I mean, so, these two were bragging charts. So, now, let's get to the actual method. And this is how it looks like. We can see some neural network here, of course, and that's actually ResNet with dilated convolutions. And so, basically, before I get into those details, there are pretty much, let me see, three parts of this pipeline. So, the first one is preparing the features that we'll need in order to predict the structure. Then the second part is the distance and torsion angle predictions, and I'll tell you what those are in a minute. So, that's the second part of pipeline, and then we have the gradient descent or the nonlinear optimization part of the pipeline. So, in stage one, they somehow find a way to go from 1D amino acid sequence that represents the protein to 2D map. And the 2D map actually contains covariations between our target sequence and similar sequence from this huge data set of sequences. And so, this is this thing called multi-sequence alignment. So, you basically find a group of sequences, like thousand sequences, which are similar to your target sequence, which you're trying to predict the structure for, and you encode the covariations in these maps. So, if I take one point on this 2D map, and this is 2D volume, so, this will be like 500 or something channels or more. There are some details down in the paper. So, one thing they encode is like the one-hot encoding of the amino acid. So, basically, that means that if we take this point, so that's... let me zoom in. Oops. So, basically, let's say this is like row five, and let's say this is like row, I don't know, 40. They'll basically encode five as a one... They'll basically find what the fifth amino acid is, and because we have only 21 amino acids in human genome, we'll encode this, let's say this is... Because there are 21, we can represent each one with a letter from the alphabet. Luckily, we have 26 letters. So, let's say this one is, I don't know, the fifth one is amino acid A, this 40th is maybe, I don't know, like S, and they just encode this one as a one-hot. Let's say this one will be like zeros, and then on some spot like 13, I don't know, they'll have 1, etc. So, what they will do is they will concatenate this one-hot vector with one-hot vector that corresponds to amino acid A, and that will be a part of the features they are using. There is much more domain-specific things they're including. Also, they're including these multi-sequence alignment things, and I'll probably explain those a bit more... I'll go into detail about those, because that's interesting a bit later. So, that's how they get this 3D volume, and now you can treat this as a simple computer vision problem, and they pretty much use this classic deep learning model like ResNet with the dialectic convolutions, and treat this as an image-to-image translation problem. So, the network itself, as you can see, has kernels of size 64 by 64, so they're just actually randomly sliding across this volume, and they end up getting the distance predictions. Okay, so what is this distance map, and how do we get one? So, in order to understand that, I have to make a small tangent here and explain how the amino acids connect, and which distances and torsion angles are we actually measuring. So, these are all the 21 amino acids we have in the human genome, and basically, as you can see, this part is always the same. For every single amino acid, no matter the properties, this part always remains the same. So, this thing here is called something called, I think, alpha carbon atom, this one is called beta carbon atom, and that one is really important. So, let's say for the sake of argument that we have a sequence like R, H, K, which is pretty much these three amino acids here. So, what will happen is, once they start going through the ribosome, they'll start connecting, and so these parts will form the backbone. So, these parts, and the groups, which is this part that goes off from the beta carbon atom, those will be, like for example, some will be positively charged, maybe this one will be negatively charged. So, what will happen is that they'll start attracting, and that's the thing that causes the protein to fold. So, imagine if you had a protein that had 140 amino acids. So, what could happen is that in this complex interaction, maybe the first amino acid would be, like in the 3D structure, would be really close to maybe 127th amino acid. So, that means we have our long range dependencies in this structure. So, now, when we are, so I mentioned distance map. So, what we are actually looking for is distance between beta carbon atoms between different amino acids. So, basically, that's what we are looking at, and then we have torsion angles, so phi and psi angles, which are basically in that 3D structure, two angles that we care about. Let me see if I can find a nice visualization of that, and you can see it here, hopefully. Let me zoom in. So, this is the protein chain, and so one of the two dihedral angles inside here, without getting into too much explanation, is two angles that we care about. So, for the sake of argument, just think there are two angles we care about. There is a third, but that one is almost always 180, so DeepMind kind of hard-coded that one to be 180, and they're just considering these two. So, okay, let me get back to the OneNote. So, now that we know this information about how we calculate distance and phi and psi angles, let's go back to the deep learning pipeline. So, basically, you can see here we have amino acids here, and we have the same sequence here, and so this particular point maybe corresponds to finding the... So, I'll actually use a different drawing, which will probably be easier to understand. Okay, let me see if I can nicely explain the distance maps. So, we have the experimentally found the pseudo ground truth distance map for this particular amino acid sequence. And we have the predicted distance map over here. So, basically, what they showed here is they took a specific amino acid from the sequence, like in particular they took 29th amino acid, and that's depicted on these charts by these red stripes. And basically, you can see that the distance between 29th carbon, beta carbon, and with itself is obviously zero, which is this colored with yellow. And over the whole main diagonal is going to be yellow for that specific, for that particular reason, because the distance between beta carbon and itself is always going to be zero. So, that's why we see this pattern along the main diagonal. And what it did here is they took the neighborhood from the 29th amino acid, and they just extracted all of the 41 distances, and that's the chart on the right. And now the thing is these things here are not scalars, they're actually probability distributions. So, if we take maybe, I don't know, 40th probability distribution, 40th amino acid and its corresponding probability distribution, we end up with something like this. And you can see on the x-axis we have the distance, and on the y-axis we have the probability. So, this particular position has, we can see that the distribution is somewhere, like the peak is pretty much around like 4 angstroms, where angstrom is 0.1 nanometer. So, this is how they symbolize the angstrom. And the red bar is the ground truth value. So, the red bar says, I don't know, like 5, and we predicted 4, which is pretty good. If we take a look at, because 29 is missing obviously, because we know that it's going to have a spike on the 0 distance, basically if we take a look at the surrounding probability distributions, so those correspond to this part here, basically distributions are really narrow and really precise, so we can see that the peak, the mode of the distribution corresponds perfectly to the red bars. Here also, really nice, as we go further apart, further away, we can see that the red bars don't correspond always with the mode of the distribution. So, say, let me take an example where we have some problems, like here, like this one is a good example, because we predicted that the distance is here, so that's whatever this value is, but the red bar is here, so, yeah. By the way, these black lines is the cut of length distance when we consider the two carbon, beta carbon atoms to be in contact. So, if we are, so this one is at 8 angstrom, so if you fall below 8 angstrom, we consider those two beta carbon atoms to be in contact, otherwise they are not in contact, obviously. So, those are basically, so the map is basically the, like the inter-residue distances for the amino sequence. And we want to predict, obviously, as close as possible to the ground truth. So, let me now go back to our initial deep learning pipeline, if I can get there, okay, we're here. So, we do a similar thing for the, for torsion angles, and once we have those maps, what we now do is we form a geometric model of our sequence, meaning we basically take, like we have a geometric model, and we have two arguments, we have the phi and we have psi, so the torsion angles, and those give us the, like, 3D coordinates of those beta carbon atoms in 3D space. So, what I mean by that, so if we have a, like a sequence of length 100, we'll have 200 parameters here. And basically, for one specific configuration of these 2L torsion angles, we get a specific configuration in 3D space, like, I don't know, like we'll have some thing going on in space. And so, what this part does is we are optimizing, so we are changing these two, so that the 3D coordinates or the corresponding 3D structure gets as close as possible to the gray one, so that this one is the native, the experimentally found structure. So, what happens is by tweaking these phi and psi angles, we are also tweaking, you can imagine, like, we are tweaking the distance map for this particular configuration, so for a given phi and psi, we have a given distance map, which we are trying to match with this one. So, the two parts of the pipeline are pretty much independent. The second pipeline, so the second part of the pipeline just figures out the depth map and the torsion maps, and then the third part of the pipeline just tries and tweaks this differentiable G model, so that we minimize the distance. And we can see on this chart here that the TM score as we are, I'm losing my cursor, my God, the TM score is going down as we are progressing with our optimization steps, and, sorry, the RMSD, that's one of the metrics that they use to figure out whether the 3D structures are, like, getting closer together, and the TM score, on the other hand, is going upwards. What's else interesting? So, they just took a snapshot here, and we can see how it's progressing and slowly getting into the final shape, which is really close to the gray one here. Now, the interesting thing is, and I didn't mention this, because those details were probably not as relevant as now, so the protein has something called the primary, the secondary, the tertiary, and the co-ordinary, like, structure. The primary structure is just the amino sequence. The secondary structure are those things that form spontaneously, like this thing, it's called alpha helix, and then we have these things here, which are called beta sheets, and then those are forming in these intricate, like, shapes, which are the tertiary structure of the protein, and finally, the co-ordinary. Let me actually use this shot, it will be easier to understand what I'm talking about. So, we have the primary structure here, then we have the secondary structures, which form, so, alpha helixes and these plebe or beta sheets, then they form in this thing called, like, the tertiary structure, and finally, multiple proteins can interact, and that's the co-ordinary structure. I hope I'm pronouncing that well. So, that's about the shapes. And aside from the distance map I already showed you, this pipeline is also predicting these secondary shapes, so the blue one is the alpha helix, and the red one is the beta sheet. So, this is the ground truth, the pseudo ground truth, and these here are taken, is a snapshot taken from the last optimization step, and we can see that the modes pretty much, we are pretty well correlated with the ground truth, and thus, the final 3D shape is also, coincides with the experimentally found shape. Whew, that was, that was, that was, that was crazy. Now, there is one more question, how do we initialize the phi's and the psi's initially? And what they did is, they used those predicted torsion angles, which are not depicted here, they only showed the distance map, and they sample from those distributions to get the initial, like, configuration, you can see it here, this is the initial one. And, it's getting crowded here. So, basically, that's how they initialize this, they randomly sample from that distribution, and then they do the optimization. But now, what they, what they, they do one more thing, that's, they keep a pool of final structures, so they keep like a 20 final structures, so whatever they get at the end of these optimization procedures, they minimize the potential, they just keep those in this pool, and then, they also sample from that pool, so they take one of these minimal configurations, and they randomly add noise to phi's and psi's, so they kind of get, experimentally show that they are getting, those are giving them better initializations than just randomly sampling from those torsion angle maps. So, that's one more detail I wanted to mention. And now, we have the whole pipeline, hopefully, in place, let me go once more, like, the high level thing, we have the features related to amino acids, we do basically convolutions over those input maps, we get the distance maps, we get the torsion angles, we get these secondary structure predictions, and once we have those, we just freeze it, and then, we have an independence step, and that's the gradient descent optimization, they used LBFGS, actually, forgot to mention that detail, and they basically are tweaking the phi's and psi's, so that the geometric model, the 3D coordinates, are getting in a configuration, which will produce distance maps, similar to the one that we got here. So, that's the high level overview of how it all fits together. Okay, that was the overview of the Alpha Fold 1, hopefully, that was useful, I know there was a lot of details, let that sink in. So, regarding the Alpha Fold 2, that's the main topic today, but we don't know much, so this is what we got from DeepMind, so we trained this system on publicly available data consisting of 170,000 protein structures from the Protein Data Bank, so we already mentioned this one, together with large bases, blah, blah, blah, and so they used 128 TPU V3 cores for maybe a couple of weeks to train this thing, and basically, this is the diagram they gave us, and the paper, so they are preparing a paper to submit for a peer-reviewed journal. And let me see what I can deduce from this chart alone. So, basically, what I expect here, and they mentioned spatial graphs in the blog, so I won't get into any assumptions aside from this one, once the paper comes out, I'll cover it, but for now, let me just try and see what we can deduce here. So, basically, what I expect to happen is, because they were using only 64 by 64 COV maps here, is that, so basically, you have the amino sequence, that's a bad depiction of amino sequence, and basically, until now, they could only model the long-range dependencies throughout the sequence, they could only take chunks, so it's basically a form of local attention, so they could maybe model, like, amino acids that lie close together, if we are on the main diagonal here, so that's, if the COV map is currently maybe here, that's when we model, like, the local attention of close-by amino acids. If the COV map is maybe here, then what happens is the following. We can model maybe this group and this group, and all the interactions between them, like this one corresponds to this one, and this one, etc. So, it becomes a mess really quick, but basically, what I expect they will do is, they will use transformers, and they'll just have, so they'll probably use some efficient transformer, a lot of them came out this year, like, I don't know, Linformer, Performer, Reformer, fill in the blank, and basically, they won't have to create these inductive biases that we currently have, and that's the intrinsic property of convolutional neural networks. Yeah, that was it about the second paper, I won't get into any more details, I just want to tell you about some really cool repercussions that the system will have in the, like, COVID-19 pandemic. So, basically, they already helped us find structures of certain proteins that are present in the SARS-CoV-2 virus. One of those is ORF3A protein, and what they can do is they can accelerate the pace of experimental methods. They can predict the structure, and then they can help the experimentation labs figure out the structure much faster than by doing it from scratch. So, yeah, that was it, hopefully you found this video insightful. If you did, go ahead and subscribe to my channel, and click that bell icon to get notified when I upload a new video, and until next time, keep learning deep.
[{"start": 0.0, "end": 6.0, "text": " It will change everything. DeepMind's AI makes a gigantic leap in solving protein structures."}, {"start": 6.0, "end": 12.0, "text": " So, I guess all of you heard about this so far, or most of you."}, {"start": 12.0, "end": 20.0, "text": " And basically what happened is that DeepMind released, published their results"}, {"start": 20.0, "end": 23.0, "text": " on this year's protein folding challenge."}, {"start": 23.0, "end": 29.0, "text": " And their method alpha fold, actually the second iteration of their algorithm alpha fold,"}, {"start": 29.0, "end": 38.0, "text": " kind of ruined the whole, like, beat everybody and got the best scores."}, {"start": 38.0, "end": 43.0, "text": " And we can see, so this is the blog that they published so far,"}, {"start": 43.0, "end": 46.0, "text": " they still haven't published the paper."}, {"start": 46.0, "end": 50.0, "text": " The paper will come probably in maybe a year or something,"}, {"start": 50.0, "end": 56.0, "text": " because the last paper for the alpha fold one came out in December 2019,"}, {"start": 56.0, "end": 59.0, "text": " where the competition happened in 2018."}, {"start": 59.0, "end": 63.0, "text": " So, we'll have to wait some time to get to that, to get the paper."}, {"start": 63.0, "end": 68.0, "text": " And I basically can see the scores throughout the years,"}, {"start": 68.0, "end": 71.0, "text": " so that's a biennial competition called CASP."}, {"start": 71.0, "end": 75.0, "text": " And it happens, so it happens every two years."}, {"start": 75.0, "end": 82.0, "text": " And you can see here that in 2018 their method alpha fold one also dominated the competition,"}, {"start": 82.0, "end": 88.0, "text": " but this year they crossed 90 on this global distance test metric,"}, {"start": 88.0, "end": 95.0, "text": " which pretty much tells you how good the alignment between the experimentally found protein structure"}, {"start": 95.0, "end": 99.0, "text": " and the computationally found protein structures are."}, {"start": 99.0, "end": 106.0, "text": " And I'll first, so the idea of this video will be to first explain you what the problem is,"}, {"start": 106.0, "end": 113.0, "text": " so what the protein folding problem is, give you a bit about history and background,"}, {"start": 113.0, "end": 115.0, "text": " and how proteins work and everything."}, {"start": 115.0, "end": 120.0, "text": " And then my idea is to go into the alpha fold one paper, because that's what we have so far,"}, {"start": 120.0, "end": 124.0, "text": " and I'll explain how it works, how the deep learning method works,"}, {"start": 124.0, "end": 128.0, "text": " so there'll be a deep, in-depth explanation of the paper."}, {"start": 128.0, "end": 135.0, "text": " And I'll give some assumptions of what will happen with the alpha fold two, what the method is,"}, {"start": 135.0, "end": 140.0, "text": " they only have one chart, so I'll try and deduce something out of it."}, {"start": 140.0, "end": 150.0, "text": " So basically, so why this is interesting is that 90 is considered as something that's like,"}, {"start": 150.0, "end": 157.0, "text": " if you get to 90 or above, you're pretty much comparable to experimental methods,"}, {"start": 157.0, "end": 163.0, "text": " the reason being they are also not perfect, and they do have their measuring uncertainties."}, {"start": 163.0, "end": 171.0, "text": " So here we can see, let me zoom in a bit, here we can see what the problem is."}, {"start": 171.0, "end": 175.0, "text": " Basically, the green thing is the pseudo ground truth,"}, {"start": 175.0, "end": 182.0, "text": " so that means somebody found that one using some of the experimental methods,"}, {"start": 182.0, "end": 184.0, "text": " and I'll tell you a bit more about those."}, {"start": 184.0, "end": 191.0, "text": " And the blue one, so the blue one is a computational prediction that DeepMind's alpha fold two made,"}, {"start": 191.0, "end": 195.0, "text": " and you can see it's pretty, pretty close."}, {"start": 195.0, "end": 200.0, "text": " The results are really, really good, and you can see 90.7, 93.3,"}, {"start": 200.0, "end": 205.0, "text": " and these are just the names of the proteins T1049, whatever."}, {"start": 205.0, "end": 212.0, "text": " So let me kind of step back a little bit here and tell you something about how the problem came to be."}, {"start": 212.0, "end": 216.0, "text": " So we've been trying to solve this problem in biology for 50 years already,"}, {"start": 216.0, "end": 223.0, "text": " and so this is kind of a really big, big thing that happened, so yeah, the hype is real this time."}, {"start": 223.0, "end": 227.0, "text": " First of all, there's this guy called Christian Anfinsen,"}, {"start": 227.0, "end": 233.0, "text": " and you can see in 1972 he gave this Nobel Prize lecture."}, {"start": 233.0, "end": 241.0, "text": " So he basically got his Nobel Prize because he showed that the function of the protein"}, {"start": 241.0, "end": 246.0, "text": " is pretty much uniquely determined by its 3D structure, which is really interesting."}, {"start": 246.0, "end": 249.0, "text": " So here is a really cool example of hemoglobin."}, {"start": 249.0, "end": 255.0, "text": " The hemoglobin forms a pocket to hold heme, a small molecule with an iron atom in the center that binds to oxygen."}, {"start": 255.0, "end": 262.0, "text": " So basically, like, the structure is made so that this protein, the hemoglobin, has a small pocket"}, {"start": 262.0, "end": 272.0, "text": " where this molecule that has, like, an iron atom which binds to, like, the oxygen and thus transports oxygen throughout our body,"}, {"start": 272.0, "end": 276.0, "text": " like, just that molecule fits so nicely in this hemoglobin."}, {"start": 276.0, "end": 282.0, "text": " So that's a really plastic example of how the structure determines the function."}, {"start": 282.0, "end": 284.0, "text": " That's a pretty obvious one."}, {"start": 284.0, "end": 287.0, "text": " So there's this cool example of this molecule that has this Y shape,"}, {"start": 287.0, "end": 295.0, "text": " and the molecule attaches to bacteria or viruses and thus stacks those cells for destruction in the immune system."}, {"start": 295.0, "end": 299.0, "text": " So let me give you just a brief example of how the proteins are created."}, {"start": 299.0, "end": 309.0, "text": " So I like to think of this as, like, a kitchen where the ribosome is basically this macromolecule acts as a cook."}, {"start": 309.0, "end": 313.0, "text": " We have the messenger RNA, which is pretty much the receipt,"}, {"start": 313.0, "end": 321.0, "text": " and then we have amino acids, which are the ingredients, which are transported by this thing called the transport RNA."}, {"start": 321.0, "end": 326.0, "text": " So basically, the messenger RNA contains this linear sequence of amino acids,"}, {"start": 326.0, "end": 332.0, "text": " and it gets translated so the receipt gets read by this ribosome,"}, {"start": 332.0, "end": 342.0, "text": " and we form, like, a chain of proteins which slowly start folding because different parts, different amino acids have different charges,"}, {"start": 342.0, "end": 346.0, "text": " and so they attract or repel, and thus we get that 3D structure."}, {"start": 346.0, "end": 353.0, "text": " So that's basically how the protein is formed, and now we're trying to figure out this 3D structure computationally."}, {"start": 353.0, "end": 362.0, "text": " Now, the thing is, how do people find the 3D structures now that we know that it's really important for, like,"}, {"start": 362.0, "end": 368.0, "text": " for figuring out the function of the proteins? How do we find the structure of the protein?"}, {"start": 368.0, "end": 374.0, "text": " So there are three methods that are used extensively in different labs,"}, {"start": 374.0, "end": 382.0, "text": " and those are mainly X-ray diffraction, nuclear magnetic resonance, and this electron microscopy."}, {"start": 382.0, "end": 392.0, "text": " And as you can see, the X-ray diffraction was so far had the highest amount of proteins were, like, protein structures were determined using this method."}, {"start": 392.0, "end": 401.0, "text": " Although it does have its own flaws, it cannot solve, so depending on the size of the protein, it has its constraints, so it can't solve everything."}, {"start": 401.0, "end": 412.0, "text": " Now, these two are getting more popular lately, but they are still super expensive and really slow methods to obtain the 3D structure."}, {"start": 412.0, "end": 419.0, "text": " So that's why the computational biology and these computational methods for figuring out the structure were so important."}, {"start": 419.0, "end": 430.0, "text": " So if we take a look at this chart here, we can see that the number of proteins we sequenced, successfully sequenced, is around 200 million now in 2020."}, {"start": 430.0, "end": 439.0, "text": " So it's been rising, like, so those are the linear sequences that are encoded in the DNA, so, but we still don't have the 3D structure for all of those."}, {"start": 439.0, "end": 444.0, "text": " And if we take a look at this chart, we can see there is this thing called a protein database,"}, {"start": 444.0, "end": 455.0, "text": " which is really important because that's the data set that's been used in the AlphaFold paper, and we'll get to it in a couple of minutes."}, {"start": 455.0, "end": 468.0, "text": " But basically here, you can see that, like, we have, so we have a lot of sequences being kind of ingested into this data set,"}, {"start": 468.0, "end": 481.0, "text": " but the number of structures is actually a lot smaller. So here, 2020, we can see only 12,000 structures are there, but we have, like, much more sequences."}, {"start": 481.0, "end": 489.0, "text": " So that was the motivation for why we want to accelerate the pace of finding 3D structures."}, {"start": 489.0, "end": 495.0, "text": " The experimental one is slow, it's really expensive, so, yeah, we need an alternative."}, {"start": 495.0, "end": 504.0, "text": " So, again, why we care about figuring out the structure, the 3D structure of the protein is because, as you can see on DeepMind's blog,"}, {"start": 504.0, "end": 513.0, "text": " an error in the genetic recipe may result in a malformed protein which could result in disease or death for an organism."}, {"start": 513.0, "end": 516.0, "text": " Many diseases, therefore, are fundamentally linked to proteins."}, {"start": 516.0, "end": 533.0, "text": " So diseases like diabetes, like dementia, Parkinson's, Alzheimer's, cancer, some of the diseases that are known to be, like, plaguing the humankind for decades or more"}, {"start": 533.0, "end": 543.0, "text": " can probably be solved by just us knowing the 3D structures. So that's why it's super important to, so that's why this problem is super important."}, {"start": 543.0, "end": 553.0, "text": " Aside from that, also, we could develop much better materials if we knew how to construct and understand the shapes of different proteins,"}, {"start": 553.0, "end": 563.0, "text": " and we could create, like, plastic-eating enzymes and thus reduce the pollution, like, on the global level, which is something we do care about, obviously."}, {"start": 563.0, "end": 571.0, "text": " So that was the background story. It was a bit longer. I assume most of my audiences, like, machine learning, have machine learning background,"}, {"start": 571.0, "end": 581.0, "text": " so I felt I need to really give you a nice overview of what the problem is and everything that goes inside, and, like, give you some numbers and methods, etc."}, {"start": 581.0, "end": 586.0, "text": " So let's start exploring the Alpha Fold 1 now."}, {"start": 586.0, "end": 597.0, "text": " As you can see here, this is an animation that DeepMind provided, which basically shows how, during their optimization procedure, which we'll get to in a sec,"}, {"start": 597.0, "end": 610.0, "text": " like, folds the proteins, so it starts from a semi-unfolded protein strand, which then slowly takes its 3D shape in space."}, {"start": 610.0, "end": 623.0, "text": " Okay, let's overview the Alpha Fold 1 paper. You can see, so, the title is Alpha Fold Improved Protein Structure Prediction Using Potentials from Deep Learning."}, {"start": 623.0, "end": 631.0, "text": " They had, like, a bunch of folks working on this project, as it's pretty common for huge projects like this one."}, {"start": 631.0, "end": 638.0, "text": " So, they started in 2016, so it's already been, like, a four-year effort of many smart people."}, {"start": 638.0, "end": 644.0, "text": " So, I've already mentioned it's DeepMind. They're based in London, and, okay."}, {"start": 644.0, "end": 649.0, "text": " So, the paper is pretty complicated. There's a lot of text and a lot of domain knowledge,"}, {"start": 649.0, "end": 658.0, "text": " so I'll just try and extract the Deep Learning side of it, and hopefully that will give you, like, a good understanding of how the method works."}, {"start": 658.0, "end": 667.0, "text": " Let's start with this chart. So, basically, on this chart here, they just show that they are better than other groups."}, {"start": 667.0, "end": 678.0, "text": " So, many other research groups were participating in this in 2018, in Cusp13 Challenge, that was the name of the challenge."}, {"start": 678.0, "end": 686.0, "text": " And, basically, the green lines represent the groups, those other groups, and the blue line is DeepMind."}, {"start": 686.0, "end": 692.0, "text": " And so, on the, basically, on the x-axis, we have something called TM score."}, {"start": 692.0, "end": 698.0, "text": " So, there are a bunch of metrics for determining how good the structure prediction is."}, {"start": 698.0, "end": 710.0, "text": " So, one is TM. There are other scores, like we saw the Global Distance Test score. There are some, like, IDDT or something."}, {"start": 710.0, "end": 721.0, "text": " Yeah, there are a lot of different scores, and basically, so, FM Domain Count, Domain is some, like, fancy name for protein, pretty much."}, {"start": 721.0, "end": 732.0, "text": " So, here, in this free modeling domain, we have 43 sequences which we needed to figure the structure for."}, {"start": 732.0, "end": 742.0, "text": " And you can see that, on average, the DeepMind had a lot higher TM score than others. So, that's what this thing is."}, {"start": 742.0, "end": 747.0, "text": " And now, I should probably tell you what the difference between free modeling and template modeling is."}, {"start": 747.0, "end": 758.0, "text": " So, template modeling is when you're trying to predict the structure for a sequence that has similar sequences whose structure we already know."}, {"start": 758.0, "end": 773.0, "text": " Now, the thing is, once you know some similar sequence, you can kind of assume that the structure of your sequence will be similar to that one."}, {"start": 773.0, "end": 783.0, "text": " So, you just need to find where the sequence differs and tweak, kind of tweak the structure, so, starting from that, maybe that structure."}, {"start": 783.0, "end": 789.0, "text": " So, that's a template modeling. So, I'm simplifying a little bit here, but, like, that's the rough notion."}, {"start": 789.0, "end": 801.0, "text": " Again, here, on this chart, they just displayed five different proteins, and they, again, show that, like, the blue dots that DeepMind is crushing it."}, {"start": 801.0, "end": 808.0, "text": " I mean, so, these two were bragging charts. So, now, let's get to the actual method."}, {"start": 808.0, "end": 819.0, "text": " And this is how it looks like. We can see some neural network here, of course, and that's actually ResNet with dilated convolutions."}, {"start": 819.0, "end": 829.0, "text": " And so, basically, before I get into those details, there are pretty much, let me see, three parts of this pipeline."}, {"start": 829.0, "end": 837.0, "text": " So, the first one is preparing the features that we'll need in order to predict the structure."}, {"start": 837.0, "end": 848.0, "text": " Then the second part is the distance and torsion angle predictions, and I'll tell you what those are in a minute."}, {"start": 848.0, "end": 856.0, "text": " So, that's the second part of pipeline, and then we have the gradient descent or the nonlinear optimization part of the pipeline."}, {"start": 856.0, "end": 868.0, "text": " So, in stage one, they somehow find a way to go from 1D amino acid sequence that represents the protein to 2D map."}, {"start": 868.0, "end": 880.0, "text": " And the 2D map actually contains covariations between our target sequence and similar sequence from this huge data set of sequences."}, {"start": 880.0, "end": 889.0, "text": " And so, this is this thing called multi-sequence alignment. So, you basically find a group of sequences, like thousand sequences,"}, {"start": 889.0, "end": 898.0, "text": " which are similar to your target sequence, which you're trying to predict the structure for, and you encode the covariations in these maps."}, {"start": 898.0, "end": 908.0, "text": " So, if I take one point on this 2D map, and this is 2D volume, so, this will be like 500 or something channels or more."}, {"start": 908.0, "end": 917.0, "text": " There are some details down in the paper. So, one thing they encode is like the one-hot encoding of the amino acid."}, {"start": 917.0, "end": 924.0, "text": " So, basically, that means that if we take this point, so that's... let me zoom in."}, {"start": 924.0, "end": 941.0, "text": " Oops. So, basically, let's say this is like row five, and let's say this is like row, I don't know, 40. They'll basically encode five as a one..."}, {"start": 941.0, "end": 955.0, "text": " They'll basically find what the fifth amino acid is, and because we have only 21 amino acids in human genome, we'll encode this, let's say this is..."}, {"start": 955.0, "end": 964.0, "text": " Because there are 21, we can represent each one with a letter from the alphabet. Luckily, we have 26 letters."}, {"start": 964.0, "end": 976.0, "text": " So, let's say this one is, I don't know, the fifth one is amino acid A, this 40th is maybe, I don't know, like S, and they just encode this one as a one-hot."}, {"start": 976.0, "end": 983.0, "text": " Let's say this one will be like zeros, and then on some spot like 13, I don't know, they'll have 1, etc."}, {"start": 983.0, "end": 998.0, "text": " So, what they will do is they will concatenate this one-hot vector with one-hot vector that corresponds to amino acid A, and that will be a part of the features they are using."}, {"start": 998.0, "end": 1011.0, "text": " There is much more domain-specific things they're including. Also, they're including these multi-sequence alignment things, and I'll probably explain those a bit more..."}, {"start": 1011.0, "end": 1015.0, "text": " I'll go into detail about those, because that's interesting a bit later."}, {"start": 1015.0, "end": 1033.0, "text": " So, that's how they get this 3D volume, and now you can treat this as a simple computer vision problem, and they pretty much use this classic deep learning model like ResNet with the dialectic convolutions,"}, {"start": 1033.0, "end": 1053.0, "text": " and treat this as an image-to-image translation problem. So, the network itself, as you can see, has kernels of size 64 by 64, so they're just actually randomly sliding across this volume, and they end up getting the distance predictions."}, {"start": 1053.0, "end": 1069.0, "text": " Okay, so what is this distance map, and how do we get one? So, in order to understand that, I have to make a small tangent here and explain how the amino acids connect, and which distances and torsion angles are we actually measuring."}, {"start": 1069.0, "end": 1089.0, "text": " So, these are all the 21 amino acids we have in the human genome, and basically, as you can see, this part is always the same. For every single amino acid, no matter the properties, this part always remains the same."}, {"start": 1089.0, "end": 1101.0, "text": " So, this thing here is called something called, I think, alpha carbon atom, this one is called beta carbon atom, and that one is really important."}, {"start": 1101.0, "end": 1124.0, "text": " So, let's say for the sake of argument that we have a sequence like R, H, K, which is pretty much these three amino acids here. So, what will happen is, once they start going through the ribosome, they'll start connecting, and so these parts will form the backbone."}, {"start": 1124.0, "end": 1143.0, "text": " So, these parts, and the groups, which is this part that goes off from the beta carbon atom, those will be, like for example, some will be positively charged, maybe this one will be negatively charged."}, {"start": 1143.0, "end": 1158.0, "text": " So, what will happen is that they'll start attracting, and that's the thing that causes the protein to fold. So, imagine if you had a protein that had 140 amino acids."}, {"start": 1158.0, "end": 1176.0, "text": " So, what could happen is that in this complex interaction, maybe the first amino acid would be, like in the 3D structure, would be really close to maybe 127th amino acid."}, {"start": 1176.0, "end": 1195.0, "text": " So, that means we have our long range dependencies in this structure. So, now, when we are, so I mentioned distance map. So, what we are actually looking for is distance between beta carbon atoms between different amino acids."}, {"start": 1195.0, "end": 1211.0, "text": " So, basically, that's what we are looking at, and then we have torsion angles, so phi and psi angles, which are basically in that 3D structure, two angles that we care about."}, {"start": 1211.0, "end": 1239.0, "text": " Let me see if I can find a nice visualization of that, and you can see it here, hopefully. Let me zoom in. So, this is the protein chain, and so one of the two dihedral angles inside here, without getting into too much explanation, is two angles that we care about."}, {"start": 1239.0, "end": 1253.0, "text": " So, for the sake of argument, just think there are two angles we care about. There is a third, but that one is almost always 180, so DeepMind kind of hard-coded that one to be 180, and they're just considering these two."}, {"start": 1253.0, "end": 1268.0, "text": " So, okay, let me get back to the OneNote. So, now that we know this information about how we calculate distance and phi and psi angles, let's go back to the deep learning pipeline."}, {"start": 1268.0, "end": 1288.0, "text": " So, basically, you can see here we have amino acids here, and we have the same sequence here, and so this particular point maybe corresponds to finding the..."}, {"start": 1288.0, "end": 1313.0, "text": " So, I'll actually use a different drawing, which will probably be easier to understand. Okay, let me see if I can nicely explain the distance maps. So, we have the experimentally found the pseudo ground truth distance map for this particular amino acid sequence."}, {"start": 1313.0, "end": 1337.0, "text": " And we have the predicted distance map over here. So, basically, what they showed here is they took a specific amino acid from the sequence, like in particular they took 29th amino acid, and that's depicted on these charts by these red stripes."}, {"start": 1337.0, "end": 1356.0, "text": " And basically, you can see that the distance between 29th carbon, beta carbon, and with itself is obviously zero, which is this colored with yellow."}, {"start": 1356.0, "end": 1369.0, "text": " And over the whole main diagonal is going to be yellow for that specific, for that particular reason, because the distance between beta carbon and itself is always going to be zero."}, {"start": 1369.0, "end": 1395.0, "text": " So, that's why we see this pattern along the main diagonal. And what it did here is they took the neighborhood from the 29th amino acid, and they just extracted all of the 41 distances, and that's the chart on the right."}, {"start": 1395.0, "end": 1418.0, "text": " And now the thing is these things here are not scalars, they're actually probability distributions. So, if we take maybe, I don't know, 40th probability distribution, 40th amino acid and its corresponding probability distribution, we end up with something like this."}, {"start": 1418.0, "end": 1444.0, "text": " And you can see on the x-axis we have the distance, and on the y-axis we have the probability. So, this particular position has, we can see that the distribution is somewhere, like the peak is pretty much around like 4 angstroms, where angstrom is 0.1 nanometer."}, {"start": 1444.0, "end": 1457.0, "text": " So, this is how they symbolize the angstrom. And the red bar is the ground truth value. So, the red bar says, I don't know, like 5, and we predicted 4, which is pretty good."}, {"start": 1457.0, "end": 1476.0, "text": " If we take a look at, because 29 is missing obviously, because we know that it's going to have a spike on the 0 distance, basically if we take a look at the surrounding probability distributions, so those correspond to this part here,"}, {"start": 1476.0, "end": 1488.0, "text": " basically distributions are really narrow and really precise, so we can see that the peak, the mode of the distribution corresponds perfectly to the red bars."}, {"start": 1488.0, "end": 1501.0, "text": " Here also, really nice, as we go further apart, further away, we can see that the red bars don't correspond always with the mode of the distribution."}, {"start": 1501.0, "end": 1521.0, "text": " So, say, let me take an example where we have some problems, like here, like this one is a good example, because we predicted that the distance is here, so that's whatever this value is, but the red bar is here, so, yeah."}, {"start": 1521.0, "end": 1537.0, "text": " By the way, these black lines is the cut of length distance when we consider the two carbon, beta carbon atoms to be in contact."}, {"start": 1537.0, "end": 1551.0, "text": " So, if we are, so this one is at 8 angstrom, so if you fall below 8 angstrom, we consider those two beta carbon atoms to be in contact, otherwise they are not in contact, obviously."}, {"start": 1551.0, "end": 1563.0, "text": " So, those are basically, so the map is basically the, like the inter-residue distances for the amino sequence."}, {"start": 1563.0, "end": 1568.0, "text": " And we want to predict, obviously, as close as possible to the ground truth."}, {"start": 1568.0, "end": 1576.0, "text": " So, let me now go back to our initial deep learning pipeline, if I can get there, okay, we're here."}, {"start": 1576.0, "end": 1590.0, "text": " So, we do a similar thing for the, for torsion angles, and once we have those maps, what we now do is we form a geometric model of our sequence,"}, {"start": 1590.0, "end": 1603.0, "text": " meaning we basically take, like we have a geometric model, and we have two arguments, we have the phi and we have psi, so the torsion angles,"}, {"start": 1603.0, "end": 1612.0, "text": " and those give us the, like, 3D coordinates of those beta carbon atoms in 3D space."}, {"start": 1612.0, "end": 1622.0, "text": " So, what I mean by that, so if we have a, like a sequence of length 100, we'll have 200 parameters here."}, {"start": 1622.0, "end": 1638.0, "text": " And basically, for one specific configuration of these 2L torsion angles, we get a specific configuration in 3D space, like, I don't know, like we'll have some thing going on in space."}, {"start": 1638.0, "end": 1658.0, "text": " And so, what this part does is we are optimizing, so we are changing these two, so that the 3D coordinates or the corresponding 3D structure gets as close as possible to the gray one,"}, {"start": 1658.0, "end": 1662.0, "text": " so that this one is the native, the experimentally found structure."}, {"start": 1662.0, "end": 1674.0, "text": " So, what happens is by tweaking these phi and psi angles, we are also tweaking, you can imagine, like, we are tweaking the distance map for this particular configuration,"}, {"start": 1674.0, "end": 1685.0, "text": " so for a given phi and psi, we have a given distance map, which we are trying to match with this one."}, {"start": 1685.0, "end": 1696.0, "text": " So, the two parts of the pipeline are pretty much independent. The second pipeline, so the second part of the pipeline just figures out the depth map and the torsion maps,"}, {"start": 1696.0, "end": 1709.0, "text": " and then the third part of the pipeline just tries and tweaks this differentiable G model, so that we minimize the distance."}, {"start": 1709.0, "end": 1723.0, "text": " And we can see on this chart here that the TM score as we are, I'm losing my cursor, my God, the TM score is going down as we are progressing with our optimization steps,"}, {"start": 1723.0, "end": 1735.0, "text": " and, sorry, the RMSD, that's one of the metrics that they use to figure out whether the 3D structures are, like, getting closer together,"}, {"start": 1735.0, "end": 1747.0, "text": " and the TM score, on the other hand, is going upwards. What's else interesting? So, they just took a snapshot here, and we can see how it's progressing"}, {"start": 1747.0, "end": 1752.0, "text": " and slowly getting into the final shape, which is really close to the gray one here."}, {"start": 1752.0, "end": 1761.0, "text": " Now, the interesting thing is, and I didn't mention this, because those details were probably not as relevant as now,"}, {"start": 1761.0, "end": 1773.0, "text": " so the protein has something called the primary, the secondary, the tertiary, and the co-ordinary, like, structure."}, {"start": 1773.0, "end": 1783.0, "text": " The primary structure is just the amino sequence. The secondary structure are those things that form spontaneously, like this thing, it's called alpha helix,"}, {"start": 1783.0, "end": 1795.0, "text": " and then we have these things here, which are called beta sheets, and then those are forming in these intricate, like, shapes,"}, {"start": 1795.0, "end": 1808.0, "text": " which are the tertiary structure of the protein, and finally, the co-ordinary. Let me actually use this shot, it will be easier to understand what I'm talking about."}, {"start": 1808.0, "end": 1817.0, "text": " So, we have the primary structure here, then we have the secondary structures, which form, so, alpha helixes and these plebe or beta sheets,"}, {"start": 1817.0, "end": 1827.0, "text": " then they form in this thing called, like, the tertiary structure, and finally, multiple proteins can interact, and that's the co-ordinary structure."}, {"start": 1827.0, "end": 1837.0, "text": " I hope I'm pronouncing that well. So, that's about the shapes. And aside from the distance map I already showed you,"}, {"start": 1837.0, "end": 1855.0, "text": " this pipeline is also predicting these secondary shapes, so the blue one is the alpha helix, and the red one is the beta sheet."}, {"start": 1855.0, "end": 1867.0, "text": " So, this is the ground truth, the pseudo ground truth, and these here are taken, is a snapshot taken from the last optimization step,"}, {"start": 1867.0, "end": 1881.0, "text": " and we can see that the modes pretty much, we are pretty well correlated with the ground truth, and thus, the final 3D shape is also,"}, {"start": 1881.0, "end": 1890.0, "text": " coincides with the experimentally found shape. Whew, that was, that was, that was, that was crazy."}, {"start": 1890.0, "end": 1896.0, "text": " Now, there is one more question, how do we initialize the phi's and the psi's initially?"}, {"start": 1896.0, "end": 1904.0, "text": " And what they did is, they used those predicted torsion angles, which are not depicted here, they only showed the distance map,"}, {"start": 1904.0, "end": 1913.0, "text": " and they sample from those distributions to get the initial, like, configuration, you can see it here, this is the initial one."}, {"start": 1913.0, "end": 1923.0, "text": " And, it's getting crowded here. So, basically, that's how they initialize this, they randomly sample from that distribution,"}, {"start": 1923.0, "end": 1934.0, "text": " and then they do the optimization. But now, what they, what they, they do one more thing, that's, they keep a pool of final structures,"}, {"start": 1934.0, "end": 1945.0, "text": " so they keep like a 20 final structures, so whatever they get at the end of these optimization procedures, they minimize the potential,"}, {"start": 1945.0, "end": 1953.0, "text": " they just keep those in this pool, and then, they also sample from that pool, so they take one of these minimal configurations,"}, {"start": 1953.0, "end": 1962.0, "text": " and they randomly add noise to phi's and psi's, so they kind of get, experimentally show that they are getting,"}, {"start": 1962.0, "end": 1970.0, "text": " those are giving them better initializations than just randomly sampling from those torsion angle maps."}, {"start": 1970.0, "end": 1979.0, "text": " So, that's one more detail I wanted to mention. And now, we have the whole pipeline, hopefully, in place, let me go once more,"}, {"start": 1979.0, "end": 1991.0, "text": " like, the high level thing, we have the features related to amino acids, we do basically convolutions over those input maps,"}, {"start": 1991.0, "end": 1997.0, "text": " we get the distance maps, we get the torsion angles, we get these secondary structure predictions,"}, {"start": 1997.0, "end": 2006.0, "text": " and once we have those, we just freeze it, and then, we have an independence step, and that's the gradient descent optimization,"}, {"start": 2006.0, "end": 2016.0, "text": " they used LBFGS, actually, forgot to mention that detail, and they basically are tweaking the phi's and psi's,"}, {"start": 2016.0, "end": 2024.0, "text": " so that the geometric model, the 3D coordinates, are getting in a configuration, which will produce distance maps,"}, {"start": 2024.0, "end": 2033.0, "text": " similar to the one that we got here. So, that's the high level overview of how it all fits together."}, {"start": 2033.0, "end": 2043.0, "text": " Okay, that was the overview of the Alpha Fold 1, hopefully, that was useful, I know there was a lot of details, let that sink in."}, {"start": 2043.0, "end": 2053.0, "text": " So, regarding the Alpha Fold 2, that's the main topic today, but we don't know much, so this is what we got from DeepMind,"}, {"start": 2053.0, "end": 2060.0, "text": " so we trained this system on publicly available data consisting of 170,000 protein structures from the Protein Data Bank,"}, {"start": 2060.0, "end": 2074.0, "text": " so we already mentioned this one, together with large bases, blah, blah, blah, and so they used 128 TPU V3 cores for maybe a couple of weeks to train this thing,"}, {"start": 2074.0, "end": 2088.0, "text": " and basically, this is the diagram they gave us, and the paper, so they are preparing a paper to submit for a peer-reviewed journal."}, {"start": 2088.0, "end": 2100.0, "text": " And let me see what I can deduce from this chart alone. So, basically, what I expect here, and they mentioned spatial graphs in the blog,"}, {"start": 2100.0, "end": 2116.0, "text": " so I won't get into any assumptions aside from this one, once the paper comes out, I'll cover it, but for now, let me just try and see what we can deduce here."}, {"start": 2116.0, "end": 2132.0, "text": " So, basically, what I expect to happen is, because they were using only 64 by 64 COV maps here, is that, so basically, you have the amino sequence,"}, {"start": 2132.0, "end": 2145.0, "text": " that's a bad depiction of amino sequence, and basically, until now, they could only model the long-range dependencies throughout the sequence,"}, {"start": 2145.0, "end": 2158.0, "text": " they could only take chunks, so it's basically a form of local attention, so they could maybe model, like, amino acids that lie close together,"}, {"start": 2158.0, "end": 2179.0, "text": " if we are on the main diagonal here, so that's, if the COV map is currently maybe here, that's when we model, like, the local attention of close-by amino acids."}, {"start": 2179.0, "end": 2195.0, "text": " If the COV map is maybe here, then what happens is the following. We can model maybe this group and this group, and all the interactions between them,"}, {"start": 2195.0, "end": 2204.0, "text": " like this one corresponds to this one, and this one, etc. So, it becomes a mess really quick, but basically, what I expect they will do is,"}, {"start": 2204.0, "end": 2215.0, "text": " they will use transformers, and they'll just have, so they'll probably use some efficient transformer, a lot of them came out this year,"}, {"start": 2215.0, "end": 2227.0, "text": " like, I don't know, Linformer, Performer, Reformer, fill in the blank, and basically, they won't have to create these inductive biases that we currently have,"}, {"start": 2227.0, "end": 2238.0, "text": " and that's the intrinsic property of convolutional neural networks. Yeah, that was it about the second paper, I won't get into any more details,"}, {"start": 2238.0, "end": 2247.0, "text": " I just want to tell you about some really cool repercussions that the system will have in the, like, COVID-19 pandemic."}, {"start": 2247.0, "end": 2262.0, "text": " So, basically, they already helped us find structures of certain proteins that are present in the SARS-CoV-2 virus."}, {"start": 2262.0, "end": 2270.0, "text": " One of those is ORF3A protein, and what they can do is they can accelerate the pace of experimental methods."}, {"start": 2270.0, "end": 2281.0, "text": " They can predict the structure, and then they can help the experimentation labs figure out the structure much faster than by doing it from scratch."}, {"start": 2281.0, "end": 2292.0, "text": " So, yeah, that was it, hopefully you found this video insightful. If you did, go ahead and subscribe to my channel,"}, {"start": 2292.0, "end": 2300.0, "text": " and click that bell icon to get notified when I upload a new video, and until next time, keep learning deep."}]