Make-A-Video: Text-To-Video Generation Without Text-Video Data | Paper Explained
🚀 Find out how to get started using Weights & Biases 🚀
http://wandb.me/ai-epiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I cover the latest text-to-video paper from Meta: "Make-A-Video: Text-To-Video Generation Without Text-Video Data".

I walk you through the 3-stage approach that consists of:
* Training a DALL-E 2 type of a model
* Integrating temporal information and tuning on unlabeled videos
* Fine-tuning the frame interpolation module.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2209.14792
✅ Website: https://makeavideo.studio/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 Intro
00:25 (sponsored) Weights & Biases
01:37 Going through the generations
06:15 High-level paper overview
10:50 Results
15:40 Limitations
16:30 Diving deep: DALL-E 2 backbone
23:35 Expanding to 3D - temporal info integration
32:39 Frame interpolation
37:24 3-stage training
41:28 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany - https://www.patreon.com/theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theaiepiphany 

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 
🐦 Twitter - https://twitter.com/gordic_aleksa 
👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE

📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/
📚 Medium - https://gordicaleksa.medium.com/ 
💻 GitHub - https://github.com/gordicaleksa 
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#makeavideo #meta #texttovideo
 What's cracking guys in this video I'm covering make a video from Meta a very cool novel Text to video model and in general over the last months. We've seen an explosion in the multimodal research So other than the text to image models such as the lead to mid-journey stable diffusion, etc Etc we've seen models that do text to audio models that do text to 3d shapes models that do text to video guys before We continue I want to make a huge shout out to weights and biases and thank them for sponsoring the video They're an amazing ML platform for tracking experiments doing data subversion in model management and much more Signing up is as simple as like literally type typing in a couple of lines of code And I want to show you one very powerful feature of weights and biases and that's reading these powerful multi media rich Reports so you can see how you can embed a video here And if you click on this particular link you end up seeing a run that produce those images Other than just producing cool videos and images you can interact with the plots as you can see here So you can see how you have a pop-up and actual numbers being displayed And if you're a power user you literally have like unlimited flexibility you can generate plots bar charts videos You can interact with your plots, and that's very cool So having said all of that there is no reason why you shouldn't check them out because they are completely free for Personal use and academic teams and they have great plans for for enterprises So if you want to support the channel sign up using the link down below and now let's get back to the video So this is not the first paper that's been doing obviously this task so we had a previous baseline such as cog video such as video diffusion Models etc etc, but this is the the one that kind of made it impressive The first one that has like really exciting results. So yeah, also there is a concurrent work I think it was called Fennaki or something that also has fastening results But I'm gonna focus on make a make a video in this one So you can see some of the videos generated here and if I open up the actual Like landing page here we can see the associated prompts So let's focus on some of them so we can see here a dog wearing a superhero outfit with red cape flying through the sky Obviously the video is kind of weird, but it's damn impressive So this is like the first the first paper that achieved results such as this. So I think it's fairly laudable So here they show some some examples of like three categories like surreal so you can see here a teddy bear painting a portrait So you can see how it looks like you can see the hands are kind of weird or something But like it's amazing that the model even had the image of a bear The imagination to generate a content like this. That's kind of cool So then they have like robot dancing in Times Square and then there is the cat watching TV with a remote in hand You can also see the hand is kind of scary But like super cool a fluffy baby sloth with an orange knitted hat Trying to figure out a laptop close-up highly detailed studio lighting screen reflecting in its eyes So so we can see the the prompt engineering culture Like perturbing all spheres of research So here is the here is the result It's fairly fascinating. I'm not sure whether this is sloth or some some some very nasty demon But it's cool nonetheless and then they have the realistic category like an artist brush painting on a canvas close-up Clown fish swimming through the coral leaf a young couple walking in in a heavy rain horse drinking water So all of these are very very cool. And then there is a stylized category. Like I'm not sure why the spaceship is stylized but Okay, because it's an alien spaceship, I guess so keep high realistic spaceship landing on the Mars An oil painting of a couple informal evening where going home get caught in a heavy downpour with umbrellas You can see that there is some artistic style here to it. You can also see a bunch of problems with the model like But like considering this is the first one and considering they also don't even have a text video Data set we'll soon see the details behind the paper. This is kind of fascinating There is a table by a window with sunlight streaming through illuminating a pile of books So again, very cool. It looks like some type of uh, like a camera beat like camera panning motion Very nice and then an emoji of a baby panda wearing a red hat So we have the red hat blue gloves so you can see that there are no blue gloves actually and then Green shirt and blue pants again some variable binding issues, but again, I will not complain Because these results are truly fascinating and considering that just like This year we we had the lead two and since then we had a cambering explosion of all of these models. So this is kind of Fairly fascinating where we've come They also can do because of the interpolation module they have which we'll see as soon as I open up the paper They can do uh, they can animate a single image so you can see here an image and then you can literally Create a video out of it. So here a bunch of different ones So again, you can see the sun kind of going through the body So the model is is is having problems with the well common sense and just uh, World understanding which is expected considering how we're training these models And then there is the turtle image here and you can also just do Two pair two images like the the source and the target image and then you can do like a smooth interpolation between the two So results look very fairly cool. There is some like blur effect here, but yeah, super cool. And also you can just do Video variation so you pick a particular video And then you generate a variety of similar videos So the this property the the the make a make a video model is inheriting pretty much from the lead to because as we'll soon see It's leveraging the lead to model as the backbone And yeah, let's let's first let's maybe dig into the paper and then we'll see those details Okay, so let me open it up here So the paper name is make a video text to video generation without text video data And I guess that's a very important point here. So there is no text video data So yeah, let's let's kind of then see how the thing works So our intuition is simple learn what the world looks like and how it is described From paired text image data And learn how the world moves. So learn how the world moves from unsupervised video footage, okay So that's going to be kind of they literally have a three-stage approach So if they first train the the text to image model, which is going to be as I said some variation of the lead to Then they they basically Train it on on the on on data sets of video data sets in an unsupervised fashion And finally, they additionally fine-tune. They train the this interpolation app work, which we'll see in a couple of minutes Okay. So as I said, it does not require paired text video data. That's kind of impressive Okay, so let's see A quick remark here So while there is a remarkable progress in the text to image generation the progress of text to video generational lags behind largely due to two main reasons So the first one is the lack of large scale data sets with high quality text video pairs I mentioned this multiple times and some of the previous work such as cog video are using Text video data sets but um to the best of my knowledge They are super small and that's the reason why they did not achieve any any significant results yet And then the second thing the second point they make here and the complexity of modeling higher dimensional video data So again, it's very compute intensive to do anything with videos. So those are two of the reasons why we didn't have any Cambrian explosion when it comes to text to video models as opposed to Text to image. Okay guys, so here is the high level architecture. I'm just gonna first just give you some Yeah, high level overview and then i'm gonna go through the experiments and then i'll later get back and dig a bit deeper into how this thing works So I want you to first recognize some components that are Same as in the delete two models if you if you haven't watched any of the delete two explanation videos Because I haven't created one i'm gonna give you a brief explanation how the model works But for now, let's just kind of identify some components So first we have input text here on the left hand side. You can see a person doing yoga outdoor during the sunrise And so what you do is you basically Convert that using a clip text encoder into a text embedding And then using this friar network, which is implemented as a diffusion model Usually then you end up with some image encoding That's going to going to well because it's trained by the clip in the background It's going to be it's going to have all of the properties of clip image embeddings and then you use that clip image embedding to condition the Diffusional model and then you use that clip image embedding to Uh condition the uh diffusion model that's going to generate so initially just ignore for the sake of argument for now Just ignore the temporal dimension just so just assume we have a single image here That's going to be the way they first train it and so this is going to be a 64 times 64 Um, like generated image from a diffusion model. Okay, then initially they don't have this frame interpolation network They just have additionally two super resolution models that do this one will do 256 256 So from 64 64 to 256 256 resolution and finally 768 or something. So yeah, so here we can see 768 and then after they've done that after they train the model in the delete two fashion What they then do is they extend these architectures the units of these architectures such that they can also integrate the temporal information then they train that whole system on the uh video data set without any labels and finally they they train this Interpolation the frame interpolation module using this masking of frames again We'll see the details, but that's kind of the rough idea that the model does learns not only the spatial Structure of the natural images that the model was trained on it also learns how to To how to model the time how the how the objects are morphing from frame to frame. That's roughly That's kind of high level intuition behind behind the thing. But yeah, that's that's it. That's how it works the three-stage approach I would say And let's quickly see the results and then i'll get back to these details behind the model So let's just kind of scheme the results so you can see on this msr vtt Benchmark, which is a video benchmark proposed by previous baselines I think cog video and other and other baselines introduced it. You can see that make a video basically, uh Like heavily outperforms them across these two metrics. So f id is way lower and also clip sim is way higher So clip sim, um, by the way, what it does is so once you generate, uh the video you basically take the frames and you see how how similar that frame is to the input text prompt that was used to generate that video so how you compute the similarity is simply a dot product between the Image embedding of the frame and text embedding of the prompt and then you just find the dot product That's the similarity and then what you do is you do average across all of the frames And that's how you get the clip sim metric to the best of my knowledge. That's how it works One thing worth mentioning is because make a video uses clip in the background, uh, that might lead to us over Estimating the actual number here or under or vice versa underestimating the numbers here because these other baselines are not Leveraging clip and thus it might be that um, yeah, we are this this number is not that maybe Accurate as this one here. So this one is kind of showing already that the model is way better compared to the baselines Cool, uh, let's continue here a couple more results. Um, they also test it on this ucf 101 in zero shot and fine tuning setting and again it outperforms The cog video which was I think either trained or fine tuned on ucf 101 training data set and it still outperforms it So that's kind of fascinating even though make a video did not use images from ucf Even though make a video did not use images from ucf. So now i'm not sure about the overlap between Lion which they are using for training make a video and ucf 101 so there might be some data pollution there But if no if not, then these results are super fascinating Uh, and also you can see in the fine tuning approach make a video again, just outperforms all of the baselines, uh heavily Okay, guys, and then what I do additionally is they do human evaluation. So they give um They they have these humans on the mech Amazone Mechanical, uh turk And basically what they do is they show side by side Here is a video from one model from our model for make a video and then the model from the baseline And they ask which one is higher quality and if it's 50 then they're kind of pretty much similar or the same So you can see here. It's always like above 50. So that means this make a video is is way better Uh, even in these uh human eval like, um experiments the faithfulness column basically tells you they ask the uh, um, those reviewers to to basically Judge how close how nicely is the generated video? Described by the text that was used to generate it and that's that's the faithfulness metric and again they're comparing Uh, they are comparing both videos side by side. Okay guys So finally before I dig into the details of the model implementation a bit deeper. Let me show you a couple more results. So Here's again the comparison between vdm So the the video diffusion models on the top We have cog video in the middle and we have the make a video on the bottom and you can again see that the two baselines here are kind of way more abstract and uh Blurry if you zoom in here the the grass doesn't have all of the like the texture information and the details Whereas this one looks uh way better just eyeballing it. It does look better compared to the baselines Then they showcased again from a static image. They generate an animated video. That's kind of cool. They here show Interpolation so this is the the target image. This is the source image and they they basically show Um how they can interpolate very nicely between those images whereas some of the baselines such as film Is struggling with understanding like some concepts like like that human hands are not supposed to to to break all of a sudden When you're interpolating between the images, so it does like some some of the um world knowledge and you can see that on the other hand This model that the make a video is is managing to model the the the The the the uh, I guess temporal has some temporal consistency and understanding I guess Cool, and finally video variation. We already saw uh actual videos on their landing page. So i'm gonna skip that one but it's uh Very very nice to have okay final thing they mentioned here, uh, the the the some of the uh downsides of the fact how they train the model and that's That they are only Using static images and text descriptions because of that. Let me read this for you So as discussed earlier our approach cannot learn associations between text and phenomenon that can only be inferred in videos So how to incorporate these along with generating longer videos with multiple scenes and events depicting more detailed stories is left for future work And obviously the reason for that is because they do not have a video text data set again mentioning that Um, so yeah things like generating a video of a person waving their hand left to right or right to left the model cannot do that reliably because Obviously static static images you cannot learn that from from a static image Okay, guys, let's now see how the model works in a bit more detail And ultimately the first thing you have to understand is how the lead to works So let me try and digest it. Obviously, it's going to be hard to explain it in a couple minutes And i'm going to refer you to some of my previous videos But yeah stick with me. Okay, so there is uh, roughly two two stages of how you train the lead to so the first one is train clip and For those of you who are not familiar with clip I already covered the paper and i've also done a code walkthrough of the original open eyes Uh repost you can check out those videos if you want to know a bit more detail about clip But on a very high level what clip does is it learns these this very rich Embedding joint embedding space between images and text such that you have this cool property where you grab a piece of text such as a corgi playing a flame throwing trumpet You basically embed it using a text encoder You get this embedding text vector And then if you do the same thing for some set of images and you do basically a dot product So you calculate the similarity between those vectors You'll find that the highest similarities exactly for those images which are best described by the text as decided by human beings so, uh, basically you can see here There would be a very high similarity between this image and this this piece of text But if I gave you an image of like a car And if I embedded the image of a car into this vector then the dot product so the similarity between that vector and this Particular text would be the similarity would be way lower and that's a nice property that clip gives us Okay, so once you have the clip And hopefully you already knew how clip works, but yeah, and and if not, hopefully the the the short description helped you Uh, okay. So then the the next uh step is to uh, basically train this model called the prior And you can see that in the original paper open ai tried a couple of different model architectures not architectures So so so basically generative models So one is the autoregressive model here and the second one is the diffusion model here and they ended up using Diffusion because it gave better results surprise surprise. Okay. So what prior does it learn it learns how to given a particular text embedding vector it learns a set of of plausible image Embedding vectors as as basically decided by clip So let me let me kind of dissect what I mean by that. Let me let me read this passage here. It's kind of important So for the diffusion prior we train a decoder only transformer. So let me first break here Let me stop here because it's decoder only so if you are familiar with diffusion models, you've mostly seen uh, unit models So the reason they're not using unit here is because we are not trying to generate an image we're trying to generate an embedding vector and so because of the the way of Because of the nature of the input and output of this prior and that's that we have a text embedding vector here and and image Embedding vector here. They're not using a unit and instead they are using a decoder only transformer. Okay, so let's continue now So we train a decoder only transformer with a causal attention mask on a sequence consisting of in order The encoded text so it's going to be the bpe encoded text the clip text embedding So that's going to be this thing here and embedding for the diffusion time step So again, that's like just the t and then embed it into some multi-dimensional space And then the noised clip image embedding Okay So it's going to be the noisy version of this one and that corresponds directly to how when you're training diffusion models on images You pass in the noisy image. So here you pass in the noisy image embedding vector, okay And finally and a final embedding whose output from the transformer is used to predict the unnoised clip image embedding Okay, so that's the idea you're kind of training get let me let me try and and draw that but uh, hopefully that was clear So you kind of have a decoder only transformer here? And you pass all of those different components which i'm not going to repeat again But like you have a bunch of these components like text embeddings blah blah blah and then ultimately you have this final embedding and on top of it you'll be generating the the unnoised version of the of the image embedding Now important thing to understand here is that the ground truth is provided by the previously trained clip model So let's assume this is the let's assume this is the oops This is the text embedding from clip then this is the image embedding of that Text image pair so these two are connected. This is like the ground truth as provided by clip And because of this it's going to learn reasonable image embeddings given a text embedding and Thus learn the prior behind the clip model if that makes sense, okay So why it's called prior is because given a text embedding vector. It's going to basically generate some plausible set of of image embedding vectors and Obviously because of the stochastic nature of diffusion models every time will generate a bit different Image embedding vector, but it's going to be Well centered around some mean of that of that output distribution. That's how that's why it's called like a prior It's setting a prior across the the possible meaningful image embeddings given that particular Text embedding. Okay. Hopefully that makes sense So once you have that then you train a decoder which is also as you can see by the diagram So so when you see these kind of dots these circles, that's a diffusion model And so what you then do is you again now you have your your regular Diffusion model training you pass in the noisy image and you try to denoise it And you just additionally condition that model throughout that process using this particular That model throughout that process using this particular Image embedding, okay And by doing that later on when you pass a purely noisy image and a particular image embedding that That diffusion model is going to generate something that that roughly corresponds to that particular image embedding Okay, hopefully that makes sense and that's how how clip works in a nutshell There is a couple more details obviously There is like two additional super resolution models because this is going to generate like a 64 by 64 image and then you need to kind of upscale it to two bigger resolutions Again, just to recap on a high level what you now do if you want to generate an image in the lead two is you grab a piece of text you embed it into the This this text embedding then you pass the text embedding into the prior prior is going to generate a reasonable well Image embedding and then you use that image embedding and you pass in a noisy image into the diffusion model To generate the 64 by 64 image and then you basically upscale it two times using the super resolution models That's it on a high level I'm not going to go any deeper than that. Okay. So now the question is how do we get the temporal part? So how do we now generate videos and the the idea is to expand the convolutions and attention layers Such that they can integrate the temporal information additionally Okay, so let me slowly start digging into those details So first just to recap the delete two part So they say here prior to the addition of the temporal components We train the backbone of our method a t2i so text to image model trained on text image pairs Sharing the core components with the work of our mesh at all. So that's the delete two paper authors Okay And there is a couple of components as I said prior network P a decoder network D and two super resolution networks S r l and and h like standing for I guess lower s and higher s. Okay guys So now let's see how the temporal piece comes into the picture They say here in order to expand the two-dimensional conditional network into the temporal dimension We modify the two key building blocks that now require not just spatial but also temporal dimensions in order to generate videos So come convolutional layers and attention layers Outer layers like fully connected don't have to be modified Okay, and then they say temporal modifications are made in most unit based diffusion networks So that means they have to modify the spatial temporal decoder dt That is now generating 16 RGB frames. Okay, so we are starting from a diffusion model That was generating a single RGB 64 by 64 image here and we are in extending it to generate 16 RGB frames, okay We're going to see the details of how that works up in a second and then not only that but they also say new They also modify the newly added frame interpolation network This upper arrow f denoted as upper arrow f increasing the effective frame rate by interpolating between the 16 Generated frames as depicted in figure 2 so and finally the super resolution network. So all of those will need to be modified They say this part so interpolating between the 16 generated frames You can see that it's going to have this this this masking logic where it's going to try to infer What what should be generated in those in those black frames and by doing that you can do either interpolation or like extrapolation Of your generated 16 frame video. That's going to be the thing with this this interpolation network. Okay So they they say a couple more details for the super resolution module So this srl so the lower s module operates across spatial and temporal dimensions In qualitative inspection we found this to significantly outperform per frame super resolution where you don't model the temporal information It is challenging to extend the srh So the high res network to the temporal dimension due to memory and compute constraints as well as the scarcity of higher resolution video data So srh operates only along the spatial dimensions But to encourage consistent detailed hallucination across frames we use the same noise initialization for each frame Okay, so that's why they have in this picture here You can see that it's this is kind of denoted like fixed noise because this one is only doing frame You're literally doing you're using the same model and just doing the the upscaling whereas this one is also actually has the temporal information integrated into the architecture Okay Okay That out of the way, let's just continue slowly and try and digest what's going on Okay, let's see how they create these pseudo 3d convolutional layers. So because Regular come 3d layers are very compute intensive They uh opt on the side of using this pseudo 3d and it's very if you're familiar with the concept of depthwise separable convolutions This does a very similar thing. So you first do spatial processing and then you do the temporal processing So you kind of decouple those as opposed to doing a joint processing of both of those Okay So it's one of the reasons of for not using the 3d Computational load and then the second reason they say is well They want to retain the previously learned spatial knowledge in the spatial convolutions weights, right? So because you first train the on the on the image text pairs you want to retain those weights and just start from there And that's what I do. They they literally do identity they have They initialize the network initially using the identity function because of that initially It's it's behaving as if you had 16 Independent 2d diffusion models. Okay. So here is how the how the process roughly looks like you grab the the input tensor which is bcfhw because you have Well see number of channels and this is the image spatial extent number of frames and batch dimension And so what I do is they first apply 2d. So it's going to be apply a spatial processing of the of the of the frames and After that they do transpose operation which is going to lead to them processing the temporal Dimension now so that's when they apply the com 1d and I think it should be 2d here as well But they're just doing like one times one, uh, like uh, uh 2d convolution basically So yeah, and then finally they also apply the the transpose operation so that they get back into this canonical shape here Okay, and as I previously mentioned note that at initialization The network will generate k different images due to random noise each faithful 2d input text But lacking any temporal coherence So let me explain why that is to understand that we need to check out this image here Okay, so here is how this is implemented So we first do the the temporal the spatial processing so you can just treat these as as as like three independent Diffusion models even though they are not but like for the sake of arguments So you first process the spatial information and then you can see here the way they initialize these temporal Filters is that you can see one here and then zeros here and because of that you you will basically end up Uh, just copy pasting the uh output so these feature maps to the next stage So initially because these are initialized the way they are you you'll just end up copy pasting these feature maps You'll just copy paste this feature map forward and then you'll continue on your processing and because of that You literally have 16 independent diffusion models that are going to generate 16 images that corresponds to correspond to the text prompt But they are very different from each other and there is no temporal consistency consistency. Okay? Temporal consistency will be a later learned using the unlabeled video data set. We'll get there a bit later Okay, so that's the first part. Hopefully that that kind of makes sense And then they have to do the same thing for the attention layers very similarly Uh, you can see here they first, um Basically do some flattening such that we have uh h times w tokens then they apply the attention which is going to be like a basically, um Complete attention pattern every token was going to attend everything every other spatial token then they do transpose that's going to Uh allow us to process that to do the same thing for the temporal extent and then again they do Uh transpose such that we are in the back in the canonical shape and then i'm flatten is going to return this into h times W if that makes sense, okay, and they say similarly to conv p3dp standing for pseudo to allow to allow for smooth Spatial temporal initialization the attention to the layer is initialized from the pre-trained T2i model and the attention 1d layer is initialized as the identity function initially, okay So let's again see the diagram here Uh, the diagrams are quite weak. Uh, and also the the code is not released So I couldn't go and dig into the code and understand some of the details that are kind of ambiguous or not clear enough from the paper But yeah, I i'll do the best I can to to to infer the the correct mechanisms here So so we first have the spatial attention So you do do the spatial attention and then you can see that this temporal attention because of the way this is initialized There are all zeros here and because of the fact we have a skipped connection That basically means that initially it's behaving as if there is no temporal Modules being added and because of that they preserve the identity the the equivalence with the text to image model So the parts that are kind of confusing is like nothing is labeled like I don't see any dimensions being being denoted here like Embedded image tokens. I'm not sure what this is referring to. It's kind of weak to be honest. Um I'm not sure whether I'll be able to reconstruct the paper from just like From just the informations that they have here. So yeah, okay guys, let's continue here There is only a couple more paragraphs worth explaining here. So let's let's focus on this frame interpolation network Okay So let's kind of read this in addition to the spatial temporal modifications discussed in the previous section. We train a new masked frame Interpolation and extrapolation network arrow up f Capable of increasing the number of frames of the generated video either by frame interpolation for a smoother generated video or by pre Post frame extrapolation for extending the video length Okay So in order to increase the frame rate within memory and compute constraints we fine tune a spatio temporal decoder On the task of masked frame interpolation by zero padding the masked input frames enabling video up sampling Okay, so this is again where the three-stage approach kicks in so they first train this spatio temporal decoder Dt using the unlabeled unsupervised Vision data set a video data set and then they they use it to to fine tune this particular model. So Um, they mentioned them here when fine tuning on masked frame interpolation We add an additional four channels to the input of the unit three channels for the rgb masked video input And an additional binary channel indicating which frames are masked Okay, so these three channels, I guess makes sense Although this is kind of in discrepancy to the drawing they have up here because this kind of makes you think that they always completely Zero out the frames and what this is kind of telling me is that they just have additional flexibility of having a Arbitrary patch inside of the image being masked out instead. So let me just go back here again So they have these four channels. So three channels For the rgb blah blah blah. So that basically means the following so you can literally grab Those three frames. Let's take one of them like maybe one that corresponds to our channel like the red channel and then you can literally mask out only a particular part of the image and then Forced the the the interpolation network to to learn what should be inside of their inside there. Okay, so um Whereas the diagram up there shows that they're constantly zeroing out the whole frames But the part that's kind of confusing to me is this binary channel indicating which frames are masked so that kind of alludes That those are either All ones or all zeros depending on whether we want to infer on whether that the that image is masked or not, but that's kind of confusing because obviously for for the Images where you don't want to do any masking you just don't do the masking and then those three channels should have enough information So i'm not sure why the fourth channel that's kind of confusing if anyone knows why feel free to comment down below Okay, so Then they say we're functioning with variable frame skips and fps so the frame uh per second Conditioning to enable multiple temporal up sample rates at inference time Okay So we denote uh blah blah blah as the operator that expands the given video tensor through mass frame interpolation Okay for all our experiments we applied This interpolation network with frameskip 5 to up sample a 16 frame video to 76 frames So here they are referring to just the eval we've seen previously where they always uh expand to this exact Frame size but like in general during the training they they allow for additional flexibility by as you can see here by by Changing the frame skips and the fps Um, they don't have to always extrapolate to this particular number to this particular number of frames So what this basically means is that what they do is they they grab an image? And then they have four images here They're going to be masked out. So these are just going to be masked out and then they have an image again and etc, etc And because they have in total 16 of the frames you can count that um, you'll have well you'll have 15 of these right you'll have 15 of these And then the last one starts ends here. So it's going to be 15 times Five plus the last one plus one and that's why they end up with 76 Okay, so that's kind of how they do it for the eval they they kind of Have these four here and then they infer what's going on in those frames And because of the way how the decoder was trained you you'll have the temporal consistency built in Okay, let's see a couple more details and then i'm going to try and make a high level overview. What's going on? So the different components of make a video described above are trained independently. Okay, that's very important So the only component that receives text as input is the prior p We train it on pair text image data and do not fine tune it on videos The decoder prior and two super resolution components are first trained on images alone. No align text Recall that the decoder receives clip image embeddings as input And the super resolution components receive down sample images as input during training Okay, so that's the dali 2 part. Okay, and now comes the second the second and the third stage So after training on images we add and initialize the new temporal layers And fine tune them over on layers and fine tune them over unlabeled video data 16 frames are sampled from the original video with random fps ranging from 1 to 30 We use beta function for sampling and while training the decoder start from higher fps ranges Which is less motion because you're taking frames with a very small temporal delta And then transition to lower fps. So that means you're you're now going to be sampling with a lot more space Time wise so so with a bigger time delta between those frames and thus you'll have more motion naturally Okay, and finally the masked frame interpolation component is fine tuned from the temporal decoder Okay, that was a mouthful. Let me try and explain this part with the video. So let's go back to the diagram above here So how I understood it Is that because now they don't have the so they don't have the text video data set because of that they what they do is They grab a video and then they sample 16 frames and they also have this curriculum learning where they initially just have This sample with a very small temporal delta and then they later have bigger and bigger temporal delta So that's kind of a form of a curriculum learning So once you have those 16 images So once you have the 16 images Because you don't have the text what you do is you just embed them using the image encoder of the clip You get the image embeddings and then you literally now Condition use those image embeddings to condition this this temporal diffusion model, right? So hopefully that makes sense. So now we're gonna do the same logic you start from a noisy image You condition each of these 16 diffusion models with with those particular image embeddings that corresponds to two particular frames And then you train it in such a fashion. So because there is this temporal integration They're not really 16 independent models But you can kind of think about it that way because you will provide them with 16 image embeddings So once you train that model the spatiotemporal decoder in that diffusion fashion Then in the third stage of the training you you grab those and you train this frame interpolation app work where you'll occasionally be masking out Well here it looks like it's like whole frame but like looking at their Description previously with the with the four additional channels. It seems they are just doing also partial Zeroing out of the frames now the part here that's confusing to me is how do you condition the the the black the black frames, right? Because in the inference time You're obviously trying to generate something that does not exist. So you do not have any conditioning signal So that means you you won't have it here either So i'm not sure how they handle there is no explanation whatsoever of how this is exactly done in the paper What they might be doing is some form of um Just msc loss or something or simply unconditional generation And because during training you actually have the ground truth images You can use some form of like msc or something to just make sure that whatever is generated is closer to the output Again confusing that doesn't make complete sense I i'm still confused a bit by this if anyone has any any hypothesis feel free to just comment down below i'll be Very happy to to read through and and and discuss what's going on here In any case guys Uh, that's it. Let me again just show you quickly the the videos. They are fairly impressive Um, I I love this one with the with the doggo It's a dog wearing a superhero outfit with red cape flying through the sky and with that i'm gonna leave you here Hopefully you took out something from this video you learned something new If you did consider subscribing to this channel share the video out and until next time. Bye. Bye guys
[{"start": 0.0, "end": 6.38, "text": " What's cracking guys in this video I'm covering make a video from Meta a very cool novel"}, {"start": 6.66, "end": 13.32, "text": " Text to video model and in general over the last months. We've seen an explosion in the multimodal research"}, {"start": 13.32, "end": 18.3, "text": " So other than the text to image models such as the lead to mid-journey stable diffusion, etc"}, {"start": 18.3, "end": 26.060000000000002, "text": " Etc we've seen models that do text to audio models that do text to 3d shapes models that do text to video guys before"}, {"start": 26.06, "end": 30.64, "text": " We continue I want to make a huge shout out to weights and biases and thank them for sponsoring the video"}, {"start": 30.96, "end": 36.62, "text": " They're an amazing ML platform for tracking experiments doing data subversion in model management and much more"}, {"start": 37.2, "end": 41.66, "text": " Signing up is as simple as like literally type typing in a couple of lines of code"}, {"start": 41.66, "end": 46.76, "text": " And I want to show you one very powerful feature of weights and biases and that's reading these powerful"}, {"start": 47.32, "end": 48.239999999999995, "text": " multi"}, {"start": 48.239999999999995, "end": 49.72, "text": " media rich"}, {"start": 49.72, "end": 52.84, "text": " Reports so you can see how you can embed a video here"}, {"start": 52.84, "end": 58.1, "text": " And if you click on this particular link you end up seeing a run that produce those images"}, {"start": 58.28, "end": 63.900000000000006, "text": " Other than just producing cool videos and images you can interact with the plots as you can see here"}, {"start": 64.4, "end": 68.88, "text": " So you can see how you have a pop-up and actual numbers being displayed"}, {"start": 69.24000000000001, "end": 77.24000000000001, "text": " And if you're a power user you literally have like unlimited flexibility you can generate plots bar charts videos"}, {"start": 77.24000000000001, "end": 80.66, "text": " You can interact with your plots, and that's very cool"}, {"start": 80.66, "end": 86.06, "text": " So having said all of that there is no reason why you shouldn't check them out because they are completely free for"}, {"start": 86.2, "end": 90.72, "text": " Personal use and academic teams and they have great plans for for enterprises"}, {"start": 90.72, "end": 96.28, "text": " So if you want to support the channel sign up using the link down below and now let's get back to the video"}, {"start": 96.44, "end": 104.34, "text": " So this is not the first paper that's been doing obviously this task so we had a previous baseline such as cog video such as video"}, {"start": 104.47999999999999, "end": 105.62, "text": " diffusion"}, {"start": 105.62, "end": 110.56, "text": " Models etc etc, but this is the the one that kind of made it impressive"}, {"start": 110.56, "end": 115.88, "text": " The first one that has like really exciting results. So yeah, also there is a concurrent work"}, {"start": 115.88, "end": 119.60000000000001, "text": " I think it was called Fennaki or something that also has fastening results"}, {"start": 119.60000000000001, "end": 123.16, "text": " But I'm gonna focus on make a make a video in this one"}, {"start": 123.56, "end": 128.62, "text": " So you can see some of the videos generated here and if I open up the actual"}, {"start": 129.08, "end": 132.52, "text": " Like landing page here we can see the associated prompts"}, {"start": 132.52, "end": 139.36, "text": " So let's focus on some of them so we can see here a dog wearing a superhero outfit with red cape flying through the sky"}, {"start": 139.36, "end": 143.04000000000002, "text": " Obviously the video is kind of weird, but it's damn impressive"}, {"start": 143.04000000000002, "end": 150.60000000000002, "text": " So this is like the first the first paper that achieved results such as this. So I think it's fairly laudable"}, {"start": 151.36, "end": 158.16000000000003, "text": " So here they show some some examples of like three categories like surreal so you can see here a teddy bear painting a portrait"}, {"start": 158.72000000000003, "end": 163.08, "text": " So you can see how it looks like you can see the hands are kind of weird or something"}, {"start": 163.08, "end": 168.4, "text": " But like it's amazing that the model even had the image of a bear"}, {"start": 168.4, "end": 172.52, "text": " The imagination to generate a content like this. That's kind of cool"}, {"start": 172.52, "end": 179.4, "text": " So then they have like robot dancing in Times Square and then there is the cat watching TV with a remote in hand"}, {"start": 179.4, "end": 181.84, "text": " You can also see the hand is kind of scary"}, {"start": 182.36, "end": 187.64000000000001, "text": " But like super cool a fluffy baby sloth with an orange knitted hat"}, {"start": 187.96, "end": 193.72, "text": " Trying to figure out a laptop close-up highly detailed studio lighting screen reflecting in its eyes"}, {"start": 193.72, "end": 196.68, "text": " So so we can see the the prompt engineering culture"}, {"start": 196.68, "end": 199.6, "text": " Like perturbing all spheres of research"}, {"start": 200.6, "end": 202.6, "text": " So here is the here is the result"}, {"start": 203.6, "end": 209.08, "text": " It's fairly fascinating. I'm not sure whether this is sloth or some some some very nasty demon"}, {"start": 209.56, "end": 216.36, "text": " But it's cool nonetheless and then they have the realistic category like an artist brush painting on a canvas close-up"}, {"start": 216.84, "end": 223.64000000000001, "text": " Clown fish swimming through the coral leaf a young couple walking in in a heavy rain horse drinking water"}, {"start": 223.64, "end": 230.44, "text": " So all of these are very very cool. And then there is a stylized category. Like I'm not sure why the spaceship is stylized but"}, {"start": 231.64, "end": 236.76, "text": " Okay, because it's an alien spaceship, I guess so keep high realistic spaceship landing on the Mars"}, {"start": 237.64, "end": 244.92, "text": " An oil painting of a couple informal evening where going home get caught in a heavy downpour with umbrellas"}, {"start": 245.56, "end": 251.56, "text": " You can see that there is some artistic style here to it. You can also see a bunch of problems with the model like"}, {"start": 251.56, "end": 257.16, "text": " But like considering this is the first one and considering they also don't even have a text video"}, {"start": 257.72, "end": 261.72, "text": " Data set we'll soon see the details behind the paper. This is kind of fascinating"}, {"start": 262.28000000000003, "end": 267.56, "text": " There is a table by a window with sunlight streaming through illuminating a pile of books"}, {"start": 268.2, "end": 273.72, "text": " So again, very cool. It looks like some type of uh, like a camera beat like camera panning motion"}, {"start": 274.28, "end": 278.68, "text": " Very nice and then an emoji of a baby panda wearing a red hat"}, {"start": 278.68, "end": 283.88, "text": " So we have the red hat blue gloves so you can see that there are no blue gloves actually and then"}, {"start": 284.28000000000003, "end": 287.88, "text": " Green shirt and blue pants again some variable binding"}, {"start": 288.76, "end": 291.24, "text": " issues, but again, I will not"}, {"start": 292.44, "end": 293.40000000000003, "text": " complain"}, {"start": 293.40000000000003, "end": 297.32, "text": " Because these results are truly fascinating and considering that just like"}, {"start": 297.8, "end": 303.24, "text": " This year we we had the lead two and since then we had a cambering explosion of all of these models. So this is kind of"}, {"start": 303.8, "end": 305.8, "text": " Fairly fascinating where we've come"}, {"start": 305.8, "end": 311.24, "text": " They also can do because of the interpolation module they have which we'll see as soon as I open up the paper"}, {"start": 311.96000000000004, "end": 317.0, "text": " They can do uh, they can animate a single image so you can see here an image and then you can literally"}, {"start": 317.56, "end": 320.52000000000004, "text": " Create a video out of it. So here a bunch of different ones"}, {"start": 320.52000000000004, "end": 322.92, "text": " So again, you can see the sun kind of going through the body"}, {"start": 322.92, "end": 328.44, "text": " So the model is is is having problems with the well common sense and just uh,"}, {"start": 329.16, "end": 333.72, "text": " World understanding which is expected considering how we're training these models"}, {"start": 333.72, "end": 336.92, "text": " And then there is the turtle image here and you can also just do"}, {"start": 337.32000000000005, "end": 343.16, "text": " Two pair two images like the the source and the target image and then you can do like a smooth interpolation between the two"}, {"start": 343.16, "end": 349.72, "text": " So results look very fairly cool. There is some like blur effect here, but yeah, super cool. And also you can just do"}, {"start": 350.76000000000005, "end": 353.72, "text": " Video variation so you pick a particular video"}, {"start": 354.52000000000004, "end": 358.36, "text": " And then you generate a variety of similar videos"}, {"start": 358.36, "end": 365.72, "text": " So the this property the the the make a make a video model is inheriting pretty much from the lead to because as we'll soon see"}, {"start": 365.72, "end": 369.32, "text": " It's leveraging the lead to model as the backbone"}, {"start": 370.12, "end": 374.52000000000004, "text": " And yeah, let's let's first let's maybe dig into the paper and then we'll see those details"}, {"start": 375.40000000000003, "end": 377.40000000000003, "text": " Okay, so let me open it up here"}, {"start": 378.36, "end": 383.32, "text": " So the paper name is make a video text to video generation without text video data"}, {"start": 383.32, "end": 388.36, "text": " And I guess that's a very important point here. So there is no text video data"}, {"start": 388.68, "end": 392.2, "text": " So yeah, let's let's kind of then see how the thing works"}, {"start": 392.36, "end": 398.04, "text": " So our intuition is simple learn what the world looks like and how it is described"}, {"start": 398.36, "end": 400.36, "text": " From paired text image data"}, {"start": 400.84, "end": 406.03999999999996, "text": " And learn how the world moves. So learn how the world moves from unsupervised video"}, {"start": 406.76, "end": 408.76, "text": " footage, okay"}, {"start": 408.76, "end": 413.0, "text": " So that's going to be kind of they literally have a three-stage approach"}, {"start": 413.0, "end": 418.68, "text": " So if they first train the the text to image model, which is going to be as I said some variation of the lead to"}, {"start": 419.48, "end": 421.48, "text": " Then they they basically"}, {"start": 421.88, "end": 427.4, "text": " Train it on on the on on data sets of video data sets in an unsupervised fashion"}, {"start": 427.64, "end": 435.32, "text": " And finally, they additionally fine-tune. They train the this interpolation app work, which we'll see in a couple of minutes"}, {"start": 435.32, "end": 439.24, "text": " Okay. So as I said, it does not require paired text video data. That's kind of impressive"}, {"start": 440.12, "end": 442.12, "text": " Okay, so let's see"}, {"start": 443.15999999999997, "end": 444.59999999999997, "text": " A quick remark here"}, {"start": 444.59999999999997, "end": 451.96, "text": " So while there is a remarkable progress in the text to image generation the progress of text to video generational lags behind largely due to two main reasons"}, {"start": 452.44, "end": 456.44, "text": " So the first one is the lack of large scale data sets with high quality text video pairs"}, {"start": 456.44, "end": 461.15999999999997, "text": " I mentioned this multiple times and some of the previous work such as cog video are using"}, {"start": 461.16, "end": 464.84000000000003, "text": " Text video data sets but um to the best of my knowledge"}, {"start": 464.84000000000003, "end": 469.56, "text": " They are super small and that's the reason why they did not achieve any any significant results yet"}, {"start": 470.36, "end": 476.44000000000005, "text": " And then the second thing the second point they make here and the complexity of modeling higher dimensional video data"}, {"start": 476.44000000000005, "end": 482.6, "text": " So again, it's very compute intensive to do anything with videos. So those are two of the reasons why we didn't have any"}, {"start": 483.40000000000003, "end": 486.68, "text": " Cambrian explosion when it comes to text to video models as opposed to"}, {"start": 486.68, "end": 492.28000000000003, "text": " Text to image. Okay guys, so here is the high level architecture. I'm just gonna first just give you some"}, {"start": 493.48, "end": 500.28000000000003, "text": " Yeah, high level overview and then i'm gonna go through the experiments and then i'll later get back and dig a bit deeper into how this thing works"}, {"start": 500.92, "end": 503.96000000000004, "text": " So I want you to first recognize some components that are"}, {"start": 504.76, "end": 511.16, "text": " Same as in the delete two models if you if you haven't watched any of the delete two explanation videos"}, {"start": 511.16, "end": 516.44, "text": " Because I haven't created one i'm gonna give you a brief explanation how the model works"}, {"start": 516.44, "end": 519.4, "text": " But for now, let's just kind of identify some components"}, {"start": 520.12, "end": 526.12, "text": " So first we have input text here on the left hand side. You can see a person doing yoga outdoor during the sunrise"}, {"start": 526.44, "end": 529.08, "text": " And so what you do is you basically"}, {"start": 530.12, "end": 535.32, "text": " Convert that using a clip text encoder into a text embedding"}, {"start": 535.32, "end": 540.6800000000001, "text": " And then using this friar network, which is implemented as a diffusion model"}, {"start": 541.48, "end": 544.9200000000001, "text": " Usually then you end up with some image encoding"}, {"start": 545.6400000000001, "end": 549.48, "text": " That's going to going to well because it's trained by the clip in the background"}, {"start": 549.48, "end": 556.6, "text": " It's going to be it's going to have all of the properties of clip image embeddings and then you use that clip image embedding to"}, {"start": 557.4000000000001, "end": 559.0, "text": " condition the"}, {"start": 559.0, "end": 562.36, "text": " Diffusional model and then you use that clip image embedding to"}, {"start": 562.36, "end": 568.76, "text": " Uh condition the uh diffusion model that's going to generate so initially just ignore for the sake of argument for now"}, {"start": 568.76, "end": 572.84, "text": " Just ignore the temporal dimension just so just assume we have a single image here"}, {"start": 573.24, "end": 577.4, "text": " That's going to be the way they first train it and so this is going to be a 64 times 64"}, {"start": 577.96, "end": 584.36, "text": " Um, like generated image from a diffusion model. Okay, then initially they don't have this frame interpolation network"}, {"start": 584.6, "end": 590.28, "text": " They just have additionally two super resolution models that do this one will do 256 256"}, {"start": 590.28, "end": 593.4, "text": " So from 64 64 to 256"}, {"start": 594.4399999999999, "end": 596.4399999999999, "text": " 256 resolution and finally"}, {"start": 597.86, "end": 599.9599999999999, "text": " 768 or something. So yeah, so here we can see"}, {"start": 600.5799999999999, "end": 605.4, "text": " 768 and then after they've done that after they train the model in the delete two fashion"}, {"start": 605.56, "end": 612.1999999999999, "text": " What they then do is they extend these architectures the units of these architectures such that they can also"}, {"start": 612.8399999999999, "end": 614.8399999999999, "text": " integrate the temporal information"}, {"start": 614.84, "end": 622.52, "text": " then they train that whole system on the uh video data set without any labels and finally they they train this"}, {"start": 623.0600000000001, "end": 628.12, "text": " Interpolation the frame interpolation module using this masking of frames again"}, {"start": 628.2, "end": 633.5600000000001, "text": " We'll see the details, but that's kind of the rough idea that the model does learns not only the spatial"}, {"start": 634.52, "end": 639.5600000000001, "text": " Structure of the natural images that the model was trained on it also learns how to"}, {"start": 639.56, "end": 645.4, "text": " To how to model the time how the how the objects are morphing from frame to frame. That's roughly"}, {"start": 645.4, "end": 649.8, "text": " That's kind of high level intuition behind behind the thing. But yeah, that's that's it. That's how it works"}, {"start": 650.52, "end": 652.52, "text": " the three-stage approach I would say"}, {"start": 653.0799999999999, "end": 658.1999999999999, "text": " And let's quickly see the results and then i'll get back to these details behind the model"}, {"start": 658.5999999999999, "end": 663.0799999999999, "text": " So let's just kind of scheme the results so you can see on this msr vtt"}, {"start": 663.4799999999999, "end": 667.0799999999999, "text": " Benchmark, which is a video benchmark proposed by previous baselines"}, {"start": 667.08, "end": 674.0400000000001, "text": " I think cog video and other and other baselines introduced it. You can see that make a video basically, uh"}, {"start": 674.9200000000001, "end": 681.48, "text": " Like heavily outperforms them across these two metrics. So f id is way lower and also clip sim is way higher"}, {"start": 681.8000000000001, "end": 688.76, "text": " So clip sim, um, by the way, what it does is so once you generate, uh the video you basically take the frames"}, {"start": 689.4000000000001, "end": 691.4000000000001, "text": " and you see how"}, {"start": 691.4000000000001, "end": 693.5600000000001, "text": " how similar that frame is to the"}, {"start": 693.56, "end": 696.8399999999999, "text": " input text prompt that was used to generate that video"}, {"start": 697.64, "end": 699.2399999999999, "text": " so how you"}, {"start": 699.2399999999999, "end": 702.52, "text": " compute the similarity is simply a dot product between the"}, {"start": 702.76, "end": 708.28, "text": " Image embedding of the frame and text embedding of the prompt and then you just find the dot product"}, {"start": 708.28, "end": 712.1199999999999, "text": " That's the similarity and then what you do is you do average across all of the frames"}, {"start": 712.4399999999999, "end": 716.68, "text": " And that's how you get the clip sim metric to the best of my knowledge. That's how it works"}, {"start": 716.68, "end": 723.7199999999999, "text": " One thing worth mentioning is because make a video uses clip in the background, uh, that might lead to us"}, {"start": 724.52, "end": 725.56, "text": " over"}, {"start": 725.56, "end": 731.9599999999999, "text": " Estimating the actual number here or under or vice versa underestimating the numbers here because these other baselines are not"}, {"start": 732.4399999999999, "end": 738.68, "text": " Leveraging clip and thus it might be that um, yeah, we are this this number is not that maybe"}, {"start": 739.0, "end": 743.9599999999999, "text": " Accurate as this one here. So this one is kind of showing already that the model is way better compared to the baselines"}, {"start": 743.96, "end": 749.74, "text": " Cool, uh, let's continue here a couple more results. Um, they also test it on this ucf"}, {"start": 750.6800000000001, "end": 755.02, "text": " 101 in zero shot and fine tuning setting and again it outperforms"}, {"start": 755.5600000000001, "end": 760.6, "text": " The cog video which was I think either trained or fine tuned on ucf"}, {"start": 761.24, "end": 765.0, "text": " 101 training data set and it still outperforms it"}, {"start": 765.0, "end": 769.48, "text": " So that's kind of fascinating even though make a video did not use images from ucf"}, {"start": 769.48, "end": 775.4, "text": " Even though make a video did not use images from ucf. So now i'm not sure about the overlap between"}, {"start": 775.72, "end": 782.9200000000001, "text": " Lion which they are using for training make a video and ucf 101 so there might be some data pollution there"}, {"start": 783.8000000000001, "end": 787.16, "text": " But if no if not, then these results are super fascinating"}, {"start": 787.8000000000001, "end": 793.72, "text": " Uh, and also you can see in the fine tuning approach make a video again, just outperforms all of the baselines, uh"}, {"start": 794.36, "end": 796.12, "text": " heavily"}, {"start": 796.12, "end": 801.88, "text": " Okay, guys, and then what I do additionally is they do human evaluation. So they give um"}, {"start": 803.0, "end": 805.12, "text": " They they have these humans on the mech"}, {"start": 806.64, "end": 807.72, "text": " Amazone Mechanical, uh turk"}, {"start": 807.72, "end": 810.12, "text": " And basically what they do is they show side by side"}, {"start": 810.2, "end": 814.84, "text": " Here is a video from one model from our model for make a video and then the model from the baseline"}, {"start": 815.16, "end": 821.72, "text": " And they ask which one is higher quality and if it's 50 then they're kind of pretty much similar or the same"}, {"start": 821.72, "end": 826.6800000000001, "text": " So you can see here. It's always like above 50. So that means this make a video is is way better"}, {"start": 827.24, "end": 831.32, "text": " Uh, even in these uh human eval like, um experiments"}, {"start": 832.2, "end": 838.9200000000001, "text": " the faithfulness column basically tells you they ask the uh, um, those reviewers to to basically"}, {"start": 839.5600000000001, "end": 843.1600000000001, "text": " Judge how close how nicely is the generated video?"}, {"start": 843.96, "end": 849.96, "text": " Described by the text that was used to generate it and that's that's the faithfulness metric and again they're comparing"}, {"start": 849.96, "end": 853.64, "text": " Uh, they are comparing both videos side by side. Okay guys"}, {"start": 853.64, "end": 860.0400000000001, "text": " So finally before I dig into the details of the model implementation a bit deeper. Let me show you a couple more results. So"}, {"start": 860.9200000000001, "end": 862.9200000000001, "text": " Here's again the comparison between vdm"}, {"start": 863.0, "end": 865.64, "text": " So the the video diffusion models on the top"}, {"start": 866.0400000000001, "end": 870.44, "text": " We have cog video in the middle and we have the make a video on the bottom and you can again see that"}, {"start": 870.84, "end": 872.84, "text": " the two baselines here are kind of"}, {"start": 873.32, "end": 875.32, "text": " way more abstract and uh"}, {"start": 875.32, "end": 880.84, "text": " Blurry if you zoom in here the the grass doesn't have all of the like the texture information and the details"}, {"start": 881.24, "end": 886.6, "text": " Whereas this one looks uh way better just eyeballing it. It does look better compared to the baselines"}, {"start": 887.24, "end": 893.8000000000001, "text": " Then they showcased again from a static image. They generate an animated video. That's kind of cool. They here show"}, {"start": 894.34, "end": 899.24, "text": " Interpolation so this is the the target image. This is the source image and they they basically show"}, {"start": 899.24, "end": 905.08, "text": " Um how they can interpolate very nicely between those images whereas some of the baselines such as film"}, {"start": 905.5600000000001, "end": 913.0, "text": " Is struggling with understanding like some concepts like like that human hands are not supposed to to to break all of a sudden"}, {"start": 913.64, "end": 921.0, "text": " When you're interpolating between the images, so it does like some some of the um world knowledge and you can see that on the other hand"}, {"start": 922.52, "end": 926.92, "text": " This model that the make a video is is managing to model the the the"}, {"start": 926.92, "end": 931.9599999999999, "text": " The the the uh, I guess temporal has some temporal consistency and understanding I guess"}, {"start": 932.8399999999999, "end": 939.3199999999999, "text": " Cool, and finally video variation. We already saw uh actual videos on their landing page. So i'm gonna skip that one but it's uh"}, {"start": 940.1999999999999, "end": 942.1999999999999, "text": " Very very nice to have okay"}, {"start": 943.0, "end": 950.28, "text": " final thing they mentioned here, uh, the the the some of the uh downsides of the fact how they train the model and that's"}, {"start": 951.4, "end": 953.0, "text": " That they are only"}, {"start": 953.0, "end": 957.56, "text": " Using static images and text descriptions because of that. Let me read this for you"}, {"start": 958.52, "end": 965.8, "text": " So as discussed earlier our approach cannot learn associations between text and phenomenon that can only be inferred in videos"}, {"start": 967.0, "end": 973.8, "text": " So how to incorporate these along with generating longer videos with multiple scenes and events depicting more detailed stories is left for future work"}, {"start": 974.12, "end": 980.2, "text": " And obviously the reason for that is because they do not have a video text data set again mentioning that"}, {"start": 980.2, "end": 984.36, "text": " Um, so yeah things like generating a video of a person"}, {"start": 984.6800000000001, "end": 988.6800000000001, "text": " waving their hand left to right or right to left the model cannot"}, {"start": 989.4000000000001, "end": 991.4000000000001, "text": " do that reliably because"}, {"start": 992.0400000000001, "end": 995.6400000000001, "text": " Obviously static static images you cannot learn that from from a static image"}, {"start": 996.12, "end": 1000.12, "text": " Okay, guys, let's now see how the model works in a bit more detail"}, {"start": 1001.1600000000001, "end": 1005.4000000000001, "text": " And ultimately the first thing you have to understand is how the lead to works"}, {"start": 1005.4, "end": 1009.9599999999999, "text": " So let me try and digest it. Obviously, it's going to be hard to explain it in a couple minutes"}, {"start": 1011.0, "end": 1013.48, "text": " And i'm going to refer you to some of my previous videos"}, {"start": 1014.1999999999999, "end": 1021.56, "text": " But yeah stick with me. Okay, so there is uh, roughly two two stages of how you train the lead to so the first one is"}, {"start": 1021.64, "end": 1023.56, "text": " train clip and"}, {"start": 1023.56, "end": 1025.8799999999999, "text": " For those of you who are not familiar with clip"}, {"start": 1025.8799999999999, "end": 1031.8799999999999, "text": " I already covered the paper and i've also done a code walkthrough of the original open eyes"}, {"start": 1031.88, "end": 1036.44, "text": " Uh repost you can check out those videos if you want to know a bit more detail about clip"}, {"start": 1037.0800000000002, "end": 1041.64, "text": " But on a very high level what clip does is it learns these this very rich"}, {"start": 1042.1200000000001, "end": 1049.5600000000002, "text": " Embedding joint embedding space between images and text such that you have this cool property where you grab a piece of text such as a corgi"}, {"start": 1049.64, "end": 1051.64, "text": " playing a flame throwing trumpet"}, {"start": 1051.96, "end": 1054.38, "text": " You basically embed it using a text encoder"}, {"start": 1054.8400000000001, "end": 1056.8400000000001, "text": " You get this embedding text vector"}, {"start": 1056.84, "end": 1061.9599999999998, "text": " And then if you do the same thing for some set of images and you do basically a dot product"}, {"start": 1061.9599999999998, "end": 1064.1999999999998, "text": " So you calculate the similarity between those vectors"}, {"start": 1064.4399999999998, "end": 1071.56, "text": " You'll find that the highest similarities exactly for those images which are best described by the text as decided by human beings"}, {"start": 1071.9599999999998, "end": 1073.9599999999998, "text": " so, uh, basically you can see here"}, {"start": 1074.36, "end": 1078.76, "text": " There would be a very high similarity between this image and this this piece of text"}, {"start": 1078.9199999999998, "end": 1080.9199999999998, "text": " But if I gave you an image of like a car"}, {"start": 1080.92, "end": 1088.2, "text": " And if I embedded the image of a car into this vector then the dot product so the similarity between that vector and this"}, {"start": 1088.3600000000001, "end": 1094.1200000000001, "text": " Particular text would be the similarity would be way lower and that's a nice property that clip gives us"}, {"start": 1094.44, "end": 1096.44, "text": " Okay, so once you have the clip"}, {"start": 1096.76, "end": 1103.8000000000002, "text": " And hopefully you already knew how clip works, but yeah, and and if not, hopefully the the the short description helped you"}, {"start": 1104.28, "end": 1109.88, "text": " Uh, okay. So then the the next uh step is to uh, basically train this model called the prior"}, {"start": 1109.88, "end": 1115.3200000000002, "text": " And you can see that in the original paper open ai tried a couple of different model"}, {"start": 1115.8600000000001, "end": 1117.3200000000002, "text": " architectures not architectures"}, {"start": 1117.3200000000002, "end": 1119.16, "text": " So so so basically generative models"}, {"start": 1119.4, "end": 1125.3200000000002, "text": " So one is the autoregressive model here and the second one is the diffusion model here and they ended up using"}, {"start": 1125.4, "end": 1132.6000000000001, "text": " Diffusion because it gave better results surprise surprise. Okay. So what prior does it learn it learns how to given a particular"}, {"start": 1132.8400000000001, "end": 1134.8400000000001, "text": " text embedding vector it learns a"}, {"start": 1135.5600000000002, "end": 1137.5600000000002, "text": " set of of plausible"}, {"start": 1137.8000000000002, "end": 1138.68, "text": " image"}, {"start": 1138.68, "end": 1142.1200000000001, "text": " Embedding vectors as as basically decided by clip"}, {"start": 1142.44, "end": 1147.16, "text": " So let me let me kind of dissect what I mean by that. Let me let me read this passage here. It's kind of important"}, {"start": 1147.72, "end": 1152.76, "text": " So for the diffusion prior we train a decoder only transformer. So let me first break here"}, {"start": 1153.16, "end": 1160.44, "text": " Let me stop here because it's decoder only so if you are familiar with diffusion models, you've mostly seen uh, unit models"}, {"start": 1160.76, "end": 1165.16, "text": " So the reason they're not using unit here is because we are not trying to generate an image"}, {"start": 1165.16, "end": 1169.64, "text": " we're trying to generate an embedding vector and so because of the the way of"}, {"start": 1170.68, "end": 1175.8000000000002, "text": " Because of the nature of the input and output of this prior and that's that we have a text embedding vector here and and image"}, {"start": 1175.8000000000002, "end": 1182.1200000000001, "text": " Embedding vector here. They're not using a unit and instead they are using a decoder only transformer. Okay, so let's continue now"}, {"start": 1182.68, "end": 1187.88, "text": " So we train a decoder only transformer with a causal attention mask on a sequence consisting of in order"}, {"start": 1188.3600000000001, "end": 1193.48, "text": " The encoded text so it's going to be the bpe encoded text the clip text embedding"}, {"start": 1193.48, "end": 1198.44, "text": " So that's going to be this thing here and embedding for the diffusion time step"}, {"start": 1198.76, "end": 1202.68, "text": " So again, that's like just the t and then embed it into some multi-dimensional space"}, {"start": 1204.28, "end": 1206.6, "text": " And then the noised clip image embedding"}, {"start": 1206.68, "end": 1206.84, "text": " Okay"}, {"start": 1206.84, "end": 1212.52, "text": " So it's going to be the noisy version of this one and that corresponds directly to how when you're training diffusion models on images"}, {"start": 1212.68, "end": 1215.88, "text": " You pass in the noisy image. So here you pass in the noisy"}, {"start": 1216.52, "end": 1218.52, "text": " image embedding vector, okay"}, {"start": 1218.52, "end": 1225.72, "text": " And finally and a final embedding whose output from the transformer is used to predict the unnoised clip image embedding"}, {"start": 1225.8799999999999, "end": 1231.32, "text": " Okay, so that's the idea you're kind of training get let me let me try and and draw that but uh, hopefully that was clear"}, {"start": 1231.32, "end": 1234.2, "text": " So you kind of have a decoder only transformer here?"}, {"start": 1235.32, "end": 1239.96, "text": " And you pass all of those different components which i'm not going to repeat again"}, {"start": 1239.96, "end": 1245.48, "text": " But like you have a bunch of these components like text embeddings blah blah blah and then ultimately you have"}, {"start": 1245.48, "end": 1253.4, "text": " this final embedding and on top of it you'll be generating the the unnoised version of the"}, {"start": 1254.1200000000001, "end": 1256.1200000000001, "text": " of the image embedding"}, {"start": 1256.68, "end": 1262.28, "text": " Now important thing to understand here is that the ground truth is provided by the previously trained clip model"}, {"start": 1262.28, "end": 1265.88, "text": " So let's assume this is the let's assume this is the oops"}, {"start": 1266.44, "end": 1272.44, "text": " This is the text embedding from clip then this is the image embedding of that"}, {"start": 1272.44, "end": 1277.88, "text": " Text image pair so these two are connected. This is like the ground truth as provided by clip"}, {"start": 1278.1200000000001, "end": 1284.04, "text": " And because of this it's going to learn reasonable image embeddings given a text embedding and"}, {"start": 1285.4, "end": 1289.64, "text": " Thus learn the prior behind the clip model if that makes sense, okay"}, {"start": 1289.64, "end": 1297.48, "text": " So why it's called prior is because given a text embedding vector. It's going to basically generate some plausible"}, {"start": 1297.48, "end": 1301.64, "text": " set of of image embedding vectors and"}, {"start": 1302.28, "end": 1306.44, "text": " Obviously because of the stochastic nature of diffusion models every time will generate a bit different"}, {"start": 1306.84, "end": 1308.84, "text": " Image embedding vector, but it's going to be"}, {"start": 1309.4, "end": 1314.28, "text": " Well centered around some mean of that of that output distribution. That's how that's why it's called like a prior"}, {"start": 1314.28, "end": 1319.88, "text": " It's setting a prior across the the possible meaningful image embeddings given that particular"}, {"start": 1320.44, "end": 1322.44, "text": " Text embedding. Okay. Hopefully that makes sense"}, {"start": 1322.44, "end": 1328.44, "text": " So once you have that then you train a decoder which is also as you can see by the diagram"}, {"start": 1328.44, "end": 1333.16, "text": " So so when you see these kind of dots these circles, that's a diffusion model"}, {"start": 1333.88, "end": 1339.0800000000002, "text": " And so what you then do is you again now you have your your regular"}, {"start": 1339.48, "end": 1343.48, "text": " Diffusion model training you pass in the noisy image and you try to denoise it"}, {"start": 1343.72, "end": 1349.3200000000002, "text": " And you just additionally condition that model throughout that process using this particular"}, {"start": 1349.32, "end": 1352.28, "text": " That model throughout that process using this particular"}, {"start": 1352.84, "end": 1354.36, "text": " Image embedding, okay"}, {"start": 1354.36, "end": 1360.28, "text": " And by doing that later on when you pass a purely noisy image and a particular image embedding that"}, {"start": 1360.9199999999998, "end": 1365.8, "text": " That diffusion model is going to generate something that that roughly corresponds to that particular image embedding"}, {"start": 1366.12, "end": 1370.2, "text": " Okay, hopefully that makes sense and that's how how clip works in a nutshell"}, {"start": 1370.6799999999998, "end": 1371.8799999999999, "text": " There is a couple more details"}, {"start": 1371.8799999999999, "end": 1372.36, "text": " obviously"}, {"start": 1372.36, "end": 1377.72, "text": " There is like two additional super resolution models because this is going to generate like a 64 by 64"}, {"start": 1377.72, "end": 1381.0, "text": " image and then you need to kind of upscale it to two bigger resolutions"}, {"start": 1381.8, "end": 1385.88, "text": " Again, just to recap on a high level what you now do if you want to generate an image"}, {"start": 1386.76, "end": 1391.08, "text": " in the lead two is you grab a piece of text you embed it into the"}, {"start": 1392.28, "end": 1399.32, "text": " This this text embedding then you pass the text embedding into the prior prior is going to generate a reasonable"}, {"start": 1399.72, "end": 1400.44, "text": " well"}, {"start": 1400.44, "end": 1405.96, "text": " Image embedding and then you use that image embedding and you pass in a noisy image into the diffusion model"}, {"start": 1405.96, "end": 1412.92, "text": " To generate the 64 by 64 image and then you basically upscale it two times using the super resolution models"}, {"start": 1413.32, "end": 1415.16, "text": " That's it on a high level"}, {"start": 1415.16, "end": 1420.6000000000001, "text": " I'm not going to go any deeper than that. Okay. So now the question is how do we get the temporal part?"}, {"start": 1420.6000000000001, "end": 1427.56, "text": " So how do we now generate videos and the the idea is to expand the convolutions and attention layers"}, {"start": 1427.96, "end": 1432.1200000000001, "text": " Such that they can integrate the temporal information additionally"}, {"start": 1432.12, "end": 1436.6, "text": " Okay, so let me slowly start digging into those details"}, {"start": 1437.2399999999998, "end": 1439.6399999999999, "text": " So first just to recap the delete two part"}, {"start": 1439.6399999999999, "end": 1442.76, "text": " So they say here prior to the addition of the temporal components"}, {"start": 1442.76, "end": 1448.1999999999998, "text": " We train the backbone of our method a t2i so text to image model trained on text image pairs"}, {"start": 1448.56, "end": 1453.0, "text": " Sharing the core components with the work of our mesh at all. So that's the delete two paper authors"}, {"start": 1453.0, "end": 1453.4799999999998, "text": " Okay"}, {"start": 1453.4799999999998, "end": 1460.76, "text": " And there is a couple of components as I said prior network P a decoder network D and two super resolution networks"}, {"start": 1460.76, "end": 1466.04, "text": " S r l and and h like standing for I guess lower s and higher s. Okay guys"}, {"start": 1466.04, "end": 1469.48, "text": " So now let's see how the temporal piece comes into the picture"}, {"start": 1470.6, "end": 1475.16, "text": " They say here in order to expand the two-dimensional conditional network into the temporal dimension"}, {"start": 1475.16, "end": 1480.92, "text": " We modify the two key building blocks that now require not just spatial but also temporal dimensions in order to generate videos"}, {"start": 1481.08, "end": 1483.72, "text": " So come convolutional layers and attention layers"}, {"start": 1484.84, "end": 1487.4, "text": " Outer layers like fully connected don't have to be modified"}, {"start": 1487.4, "end": 1492.68, "text": " Okay, and then they say temporal modifications are made in most unit based diffusion networks"}, {"start": 1493.3200000000002, "end": 1497.88, "text": " So that means they have to modify the spatial temporal decoder dt"}, {"start": 1498.3600000000001, "end": 1503.5600000000002, "text": " That is now generating 16 RGB frames. Okay, so we are starting from a diffusion model"}, {"start": 1503.5600000000002, "end": 1511.0, "text": " That was generating a single RGB 64 by 64 image here and we are in extending it to generate"}, {"start": 1511.52, "end": 1513.52, "text": " 16 RGB frames, okay"}, {"start": 1513.52, "end": 1519.12, "text": " We're going to see the details of how that works up in a second and then not only that but they also say new"}, {"start": 1519.12, "end": 1522.24, "text": " They also modify the newly added frame interpolation network"}, {"start": 1523.12, "end": 1529.28, "text": " This upper arrow f denoted as upper arrow f increasing the effective frame rate by interpolating between the 16"}, {"start": 1529.84, "end": 1536.4, "text": " Generated frames as depicted in figure 2 so and finally the super resolution network. So all of those will need to be modified"}, {"start": 1536.8799999999999, "end": 1540.6399999999999, "text": " They say this part so interpolating between the 16 generated frames"}, {"start": 1540.64, "end": 1546.0800000000002, "text": " You can see that it's going to have this this this masking logic where it's going to try to infer"}, {"start": 1546.4, "end": 1553.44, "text": " What what should be generated in those in those black frames and by doing that you can do either interpolation or like extrapolation"}, {"start": 1554.24, "end": 1561.2800000000002, "text": " Of your generated 16 frame video. That's going to be the thing with this this interpolation network. Okay"}, {"start": 1562.24, "end": 1566.88, "text": " So they they say a couple more details for the super resolution module"}, {"start": 1566.88, "end": 1572.88, "text": " So this srl so the lower s module operates across spatial and temporal dimensions"}, {"start": 1573.68, "end": 1580.4, "text": " In qualitative inspection we found this to significantly outperform per frame super resolution where you don't model the temporal information"}, {"start": 1580.8000000000002, "end": 1583.0400000000002, "text": " It is challenging to extend the srh"}, {"start": 1583.0400000000002, "end": 1589.8400000000001, "text": " So the high res network to the temporal dimension due to memory and compute constraints as well as the scarcity of higher resolution video data"}, {"start": 1589.8400000000001, "end": 1594.0, "text": " So srh operates only along the spatial dimensions"}, {"start": 1594.0, "end": 1600.08, "text": " But to encourage consistent detailed hallucination across frames we use the same noise initialization for each frame"}, {"start": 1600.08, "end": 1602.64, "text": " Okay, so that's why they have in this picture here"}, {"start": 1602.64, "end": 1608.32, "text": " You can see that it's this is kind of denoted like fixed noise because this one is only doing frame"}, {"start": 1608.32, "end": 1616.72, "text": " You're literally doing you're using the same model and just doing the the upscaling whereas this one is also actually has the temporal information"}, {"start": 1617.76, "end": 1619.76, "text": " integrated into the architecture"}, {"start": 1620.64, "end": 1622.64, "text": " Okay"}, {"start": 1622.64, "end": 1624.48, "text": " Okay"}, {"start": 1624.48, "end": 1628.64, "text": " That out of the way, let's just continue slowly and try and digest what's going on"}, {"start": 1629.76, "end": 1635.2, "text": " Okay, let's see how they create these pseudo 3d convolutional layers. So because"}, {"start": 1636.3000000000002, "end": 1640.42, "text": " Regular come 3d layers are very compute intensive"}, {"start": 1640.8000000000002, "end": 1648.26, "text": " They uh opt on the side of using this pseudo 3d and it's very if you're familiar with the concept of depthwise separable convolutions"}, {"start": 1648.26, "end": 1653.3799999999999, "text": " This does a very similar thing. So you first do spatial processing and then you do the temporal processing"}, {"start": 1653.3799999999999, "end": 1657.62, "text": " So you kind of decouple those as opposed to doing a joint processing of both of those"}, {"start": 1657.86, "end": 1658.82, "text": " Okay"}, {"start": 1658.82, "end": 1661.3, "text": " So it's one of the reasons of for not using the 3d"}, {"start": 1661.84, "end": 1664.82, "text": " Computational load and then the second reason they say is well"}, {"start": 1664.9, "end": 1669.46, "text": " They want to retain the previously learned spatial knowledge in the spatial convolutions weights, right?"}, {"start": 1669.62, "end": 1676.34, "text": " So because you first train the on the on the image text pairs you want to retain those weights and just start from there"}, {"start": 1676.34, "end": 1679.6999999999998, "text": " And that's what I do. They they literally do identity"}, {"start": 1680.58, "end": 1682.1, "text": " they have"}, {"start": 1682.1, "end": 1685.9399999999998, "text": " They initialize the network initially using the identity function because of that initially"}, {"start": 1686.02, "end": 1688.74, "text": " It's it's behaving as if you had 16"}, {"start": 1689.52, "end": 1696.58, "text": " Independent 2d diffusion models. Okay. So here is how the how the process roughly looks like you grab the"}, {"start": 1697.62, "end": 1699.62, "text": " the input tensor which is"}, {"start": 1700.6399999999999, "end": 1702.6399999999999, "text": " bcfhw because you have"}, {"start": 1702.64, "end": 1708.8000000000002, "text": " Well see number of channels and this is the image spatial extent number of frames and batch dimension"}, {"start": 1709.0400000000002, "end": 1713.92, "text": " And so what I do is they first apply 2d. So it's going to be apply a spatial"}, {"start": 1714.46, "end": 1717.6000000000001, "text": " processing of the of the of the frames and"}, {"start": 1718.24, "end": 1723.92, "text": " After that they do transpose operation which is going to lead to them processing the temporal"}, {"start": 1724.48, "end": 1729.6000000000001, "text": " Dimension now so that's when they apply the com 1d and I think it should be 2d here as well"}, {"start": 1729.6, "end": 1735.36, "text": " But they're just doing like one times one, uh, like uh, uh 2d convolution basically"}, {"start": 1735.4399999999998, "end": 1741.9199999999998, "text": " So yeah, and then finally they also apply the the transpose operation so that they get back into this canonical"}, {"start": 1742.56, "end": 1744.32, "text": " shape here"}, {"start": 1744.32, "end": 1747.4599999999998, "text": " Okay, and as I previously mentioned note that at initialization"}, {"start": 1748.24, "end": 1754.9599999999998, "text": " The network will generate k different images due to random noise each faithful 2d input text"}, {"start": 1754.9599999999998, "end": 1756.9599999999998, "text": " But lacking any temporal coherence"}, {"start": 1756.96, "end": 1762.56, "text": " So let me explain why that is to understand that we need to check out this image here"}, {"start": 1762.8, "end": 1764.8, "text": " Okay, so here is how this is implemented"}, {"start": 1764.88, "end": 1772.1000000000001, "text": " So we first do the the temporal the spatial processing so you can just treat these as as as like three independent"}, {"start": 1772.72, "end": 1776.16, "text": " Diffusion models even though they are not but like for the sake of arguments"}, {"start": 1776.24, "end": 1782.24, "text": " So you first process the spatial information and then you can see here the way they initialize these temporal"}, {"start": 1782.24, "end": 1790.08, "text": " Filters is that you can see one here and then zeros here and because of that you you will basically end up"}, {"start": 1790.24, "end": 1795.68, "text": " Uh, just copy pasting the uh output so these feature maps to the next stage"}, {"start": 1795.68, "end": 1802.48, "text": " So initially because these are initialized the way they are you you'll just end up copy pasting these feature maps"}, {"start": 1802.48, "end": 1808.16, "text": " You'll just copy paste this feature map forward and then you'll continue on your processing and because of that"}, {"start": 1808.16, "end": 1816.0, "text": " You literally have 16 independent diffusion models that are going to generate 16 images that corresponds to correspond to the text prompt"}, {"start": 1816.16, "end": 1820.8000000000002, "text": " But they are very different from each other and there is no temporal consistency consistency. Okay?"}, {"start": 1821.68, "end": 1828.16, "text": " Temporal consistency will be a later learned using the unlabeled video data set. We'll get there a bit later"}, {"start": 1828.3200000000002, "end": 1831.52, "text": " Okay, so that's the first part. Hopefully that that kind of makes sense"}, {"start": 1832.24, "end": 1836.16, "text": " And then they have to do the same thing for the attention layers very similarly"}, {"start": 1836.16, "end": 1838.88, "text": " Uh, you can see here they first, um"}, {"start": 1839.76, "end": 1846.72, "text": " Basically do some flattening such that we have uh h times w tokens then they apply the attention which is going to be like a"}, {"start": 1847.0400000000002, "end": 1848.96, "text": " basically, um"}, {"start": 1848.96, "end": 1855.2, "text": " Complete attention pattern every token was going to attend everything every other spatial token then they do transpose that's going to"}, {"start": 1855.3600000000001, "end": 1860.72, "text": " Uh allow us to process that to do the same thing for the temporal extent and then again they do"}, {"start": 1860.72, "end": 1867.2, "text": " Uh transpose such that we are in the back in the canonical shape and then i'm flatten is going to return this into h"}, {"start": 1868.48, "end": 1870.08, "text": " times"}, {"start": 1870.08, "end": 1878.16, "text": " W if that makes sense, okay, and they say similarly to conv p3dp standing for pseudo to allow to allow for smooth"}, {"start": 1878.8, "end": 1883.28, "text": " Spatial temporal initialization the attention to the layer is initialized from the pre-trained"}, {"start": 1883.6000000000001, "end": 1889.2, "text": " T2i model and the attention 1d layer is initialized as the identity function initially, okay"}, {"start": 1889.2, "end": 1891.76, "text": " So let's again see the diagram here"}, {"start": 1892.4, "end": 1896.48, "text": " Uh, the diagrams are quite weak. Uh, and also the the code is not released"}, {"start": 1896.48, "end": 1902.32, "text": " So I couldn't go and dig into the code and understand some of the details that are kind of ambiguous or not clear enough from the paper"}, {"start": 1902.32, "end": 1908.24, "text": " But yeah, I i'll do the best I can to to to infer the the correct mechanisms here"}, {"start": 1909.04, "end": 1911.3600000000001, "text": " So so we first have the spatial attention"}, {"start": 1911.3600000000001, "end": 1917.8400000000001, "text": " So you do do the spatial attention and then you can see that this temporal attention because of the way this is initialized"}, {"start": 1917.84, "end": 1922.32, "text": " There are all zeros here and because of the fact we have a skipped connection"}, {"start": 1922.8799999999999, "end": 1928.08, "text": " That basically means that initially it's behaving as if there is no temporal"}, {"start": 1928.9599999999998, "end": 1935.1999999999998, "text": " Modules being added and because of that they preserve the identity the the equivalence with the text to image"}, {"start": 1935.84, "end": 1937.4399999999998, "text": " model"}, {"start": 1937.4399999999998, "end": 1943.6, "text": " So the parts that are kind of confusing is like nothing is labeled like I don't see any dimensions being being denoted here like"}, {"start": 1943.6, "end": 1950.8, "text": " Embedded image tokens. I'm not sure what this is referring to. It's kind of weak to be honest. Um"}, {"start": 1952.1599999999999, "end": 1955.4399999999998, "text": " I'm not sure whether I'll be able to reconstruct the paper from just like"}, {"start": 1956.48, "end": 1961.12, "text": " From just the informations that they have here. So yeah, okay guys, let's continue here"}, {"start": 1961.1999999999998, "end": 1967.6799999999998, "text": " There is only a couple more paragraphs worth explaining here. So let's let's focus on this frame interpolation network"}, {"start": 1967.76, "end": 1968.6399999999999, "text": " Okay"}, {"start": 1968.64, "end": 1974.8000000000002, "text": " So let's kind of read this in addition to the spatial temporal modifications discussed in the previous section. We train a new masked frame"}, {"start": 1975.1000000000001, "end": 1977.92, "text": " Interpolation and extrapolation network arrow up f"}, {"start": 1978.5600000000002, "end": 1986.16, "text": " Capable of increasing the number of frames of the generated video either by frame interpolation for a smoother generated video or by pre"}, {"start": 1986.24, "end": 1988.5600000000002, "text": " Post frame extrapolation for extending the video length"}, {"start": 1988.5600000000002, "end": 1989.2, "text": " Okay"}, {"start": 1989.2, "end": 1995.68, "text": " So in order to increase the frame rate within memory and compute constraints we fine tune a spatio temporal decoder"}, {"start": 1995.68, "end": 2003.76, "text": " On the task of masked frame interpolation by zero padding the masked input frames enabling video up sampling"}, {"start": 2004.64, "end": 2010.3200000000002, "text": " Okay, so this is again where the three-stage approach kicks in so they first train this spatio temporal decoder"}, {"start": 2010.64, "end": 2013.78, "text": " Dt using the unlabeled unsupervised"}, {"start": 2014.4, "end": 2021.2, "text": " Vision data set a video data set and then they they use it to to fine tune this particular model. So"}, {"start": 2021.2, "end": 2026.4, "text": " Um, they mentioned them here when fine tuning on masked frame interpolation"}, {"start": 2026.4, "end": 2033.68, "text": " We add an additional four channels to the input of the unit three channels for the rgb masked video input"}, {"start": 2034.4, "end": 2037.3600000000001, "text": " And an additional binary channel indicating which frames are masked"}, {"start": 2037.92, "end": 2040.48, "text": " Okay, so these three channels, I guess makes sense"}, {"start": 2040.56, "end": 2048.08, "text": " Although this is kind of in discrepancy to the drawing they have up here because this kind of makes you think that they always"}, {"start": 2048.7, "end": 2050.08, "text": " completely"}, {"start": 2050.08, "end": 2058.16, "text": " Zero out the frames and what this is kind of telling me is that they just have additional flexibility of having a"}, {"start": 2058.54, "end": 2064.0, "text": " Arbitrary patch inside of the image being masked out instead. So let me just go back here again"}, {"start": 2065.04, "end": 2067.7599999999998, "text": " So they have these four channels. So three channels"}, {"start": 2068.4, "end": 2073.2799999999997, "text": " For the rgb blah blah blah. So that basically means the following so you can literally"}, {"start": 2074.48, "end": 2075.44, "text": " grab"}, {"start": 2075.44, "end": 2080.88, "text": " Those three frames. Let's take one of them like maybe one that corresponds to our channel like the red channel"}, {"start": 2081.12, "end": 2085.2000000000003, "text": " and then you can literally mask out only a particular part of the image and then"}, {"start": 2086.08, "end": 2093.12, "text": " Forced the the the interpolation network to to learn what should be inside of their inside there. Okay, so um"}, {"start": 2093.6, "end": 2098.48, "text": " Whereas the diagram up there shows that they're constantly zeroing out the whole frames"}, {"start": 2099.04, "end": 2103.76, "text": " But the part that's kind of confusing to me is this binary channel indicating which frames are masked"}, {"start": 2103.76, "end": 2106.0800000000004, "text": " so that kind of alludes"}, {"start": 2107.28, "end": 2109.28, "text": " That those are either"}, {"start": 2110.0800000000004, "end": 2113.36, "text": " All ones or all zeros depending on whether we want to"}, {"start": 2114.0800000000004, "end": 2115.6800000000003, "text": " infer"}, {"start": 2115.6800000000003, "end": 2120.32, "text": " on whether that the that image is masked or not, but that's kind of confusing because"}, {"start": 2121.1200000000003, "end": 2123.1200000000003, "text": " obviously for for the"}, {"start": 2123.1200000000003, "end": 2128.96, "text": " Images where you don't want to do any masking you just don't do the masking and then those three channels should have enough information"}, {"start": 2128.96, "end": 2136.08, "text": " So i'm not sure why the fourth channel that's kind of confusing if anyone knows why feel free to comment down below"}, {"start": 2136.56, "end": 2138.56, "text": " Okay, so"}, {"start": 2138.64, "end": 2144.0, "text": " Then they say we're functioning with variable frame skips and fps so the frame uh per second"}, {"start": 2145.1, "end": 2149.36, "text": " Conditioning to enable multiple temporal up sample rates at inference time"}, {"start": 2150.0, "end": 2151.28, "text": " Okay"}, {"start": 2151.28, "end": 2156.96, "text": " So we denote uh blah blah blah as the operator that expands the given video tensor through mass frame interpolation"}, {"start": 2156.96, "end": 2159.12, "text": " Okay for all our experiments we applied"}, {"start": 2159.76, "end": 2165.68, "text": " This interpolation network with frameskip 5 to up sample a 16 frame video to 76 frames"}, {"start": 2165.68, "end": 2171.68, "text": " So here they are referring to just the eval we've seen previously where they always uh expand to this exact"}, {"start": 2172.32, "end": 2178.8, "text": " Frame size but like in general during the training they they allow for additional flexibility by as you can see here by by"}, {"start": 2179.04, "end": 2181.04, "text": " Changing the frame skips and the fps"}, {"start": 2181.04, "end": 2186.4, "text": " Um, they don't have to always extrapolate to this particular number to this particular number of frames"}, {"start": 2186.88, "end": 2191.04, "text": " So what this basically means is that what they do is they they grab an image?"}, {"start": 2192.4, "end": 2194.4, "text": " And then they have four images here"}, {"start": 2194.96, "end": 2200.16, "text": " They're going to be masked out. So these are just going to be masked out and then they have an image again"}, {"start": 2200.8, "end": 2202.8, "text": " and etc, etc"}, {"start": 2202.8, "end": 2205.7599999999998, "text": " And because they have in total 16 of the frames"}, {"start": 2206.4, "end": 2209.52, "text": " you can count that um, you'll have well you'll have"}, {"start": 2209.52, "end": 2212.88, "text": " 15 of these right you'll have 15 of these"}, {"start": 2213.68, "end": 2217.28, "text": " And then the last one starts ends here. So it's going to be 15"}, {"start": 2218.08, "end": 2219.52, "text": " times"}, {"start": 2219.52, "end": 2224.24, "text": " Five plus the last one plus one and that's why they end up with 76"}, {"start": 2225.12, "end": 2230.0, "text": " Okay, so that's kind of how they do it for the eval they they kind of"}, {"start": 2230.56, "end": 2234.24, "text": " Have these four here and then they infer what's going on in those frames"}, {"start": 2234.24, "end": 2240.08, "text": " And because of the way how the decoder was trained you you'll have the temporal consistency built in"}, {"start": 2240.56, "end": 2245.2, "text": " Okay, let's see a couple more details and then i'm going to try and make a high level overview. What's going on?"}, {"start": 2246.08, "end": 2251.2799999999997, "text": " So the different components of make a video described above are trained independently. Okay, that's very important"}, {"start": 2251.8399999999997, "end": 2255.2799999999997, "text": " So the only component that receives text as input is the prior p"}, {"start": 2255.6, "end": 2258.8799999999997, "text": " We train it on pair text image data and do not fine tune it on videos"}, {"start": 2258.88, "end": 2264.56, "text": " The decoder prior and two super resolution components are first trained on images alone. No align text"}, {"start": 2264.96, "end": 2268.32, "text": " Recall that the decoder receives clip image embeddings as input"}, {"start": 2268.7200000000003, "end": 2272.88, "text": " And the super resolution components receive down sample images as input during training"}, {"start": 2273.12, "end": 2277.92, "text": " Okay, so that's the dali 2 part. Okay, and now comes the second the second and the third stage"}, {"start": 2278.1600000000003, "end": 2282.48, "text": " So after training on images we add and initialize the new temporal layers"}, {"start": 2283.36, "end": 2285.76, "text": " And fine tune them over on layers"}, {"start": 2285.76, "end": 2288.8, "text": " and fine tune them over unlabeled video data"}, {"start": 2289.6000000000004, "end": 2296.1600000000003, "text": " 16 frames are sampled from the original video with random fps ranging from 1 to 30"}, {"start": 2297.0400000000004, "end": 2303.0400000000004, "text": " We use beta function for sampling and while training the decoder start from higher fps ranges"}, {"start": 2304.0, "end": 2309.0400000000004, "text": " Which is less motion because you're taking frames with a very small temporal delta"}, {"start": 2309.04, "end": 2315.12, "text": " And then transition to lower fps. So that means you're you're now going to be sampling with a lot more space"}, {"start": 2315.68, "end": 2321.84, "text": " Time wise so so with a bigger time delta between those frames and thus you'll have more motion naturally"}, {"start": 2321.84, "end": 2326.4, "text": " Okay, and finally the masked frame interpolation component is fine tuned from the temporal decoder"}, {"start": 2327.52, "end": 2334.48, "text": " Okay, that was a mouthful. Let me try and explain this part with the video. So let's go back to the diagram above here"}, {"start": 2335.12, "end": 2337.12, "text": " So how I understood it"}, {"start": 2337.12, "end": 2342.24, "text": " Is that because now they don't have the so they don't have the text video"}, {"start": 2342.88, "end": 2346.24, "text": " data set because of that they what they do is"}, {"start": 2346.72, "end": 2353.12, "text": " They grab a video and then they sample 16 frames and they also have this curriculum learning where they initially just have"}, {"start": 2353.6, "end": 2359.92, "text": " This sample with a very small temporal delta and then they later have bigger and bigger temporal delta"}, {"start": 2359.92, "end": 2361.92, "text": " So that's kind of a form of a curriculum learning"}, {"start": 2362.56, "end": 2364.56, "text": " So once you have those 16 images"}, {"start": 2364.56, "end": 2366.56, "text": " So once you have the 16 images"}, {"start": 2367.04, "end": 2372.7999999999997, "text": " Because you don't have the text what you do is you just embed them using the image encoder of the clip"}, {"start": 2373.52, "end": 2378.08, "text": " You get the image embeddings and then you literally now"}, {"start": 2378.86, "end": 2384.56, "text": " Condition use those image embeddings to condition this this temporal diffusion model, right?"}, {"start": 2384.88, "end": 2389.12, "text": " So hopefully that makes sense. So now we're gonna do the same logic you start from a noisy image"}, {"start": 2390.0, "end": 2392.24, "text": " You condition each of these 16"}, {"start": 2392.24, "end": 2400.0, "text": " diffusion models with with those particular image embeddings that corresponds to two particular frames"}, {"start": 2401.04, "end": 2405.12, "text": " And then you train it in such a fashion. So because there is this temporal integration"}, {"start": 2405.12, "end": 2407.4399999999996, "text": " They're not really 16 independent models"}, {"start": 2407.6, "end": 2412.72, "text": " But you can kind of think about it that way because you will provide them with 16 image embeddings"}, {"start": 2413.3599999999997, "end": 2416.5, "text": " So once you train that model the spatiotemporal decoder"}, {"start": 2417.2799999999997, "end": 2419.2799999999997, "text": " in that diffusion fashion"}, {"start": 2419.28, "end": 2426.48, "text": " Then in the third stage of the training you you grab those and you train this frame interpolation app work where you'll occasionally"}, {"start": 2427.2000000000003, "end": 2429.2000000000003, "text": " be masking out"}, {"start": 2429.44, "end": 2432.8, "text": " Well here it looks like it's like whole frame but like looking at their"}, {"start": 2433.6800000000003, "end": 2439.6000000000004, "text": " Description previously with the with the four additional channels. It seems they are just doing also partial"}, {"start": 2440.96, "end": 2448.0, "text": " Zeroing out of the frames now the part here that's confusing to me is how do you condition the the the black the black frames, right?"}, {"start": 2448.0, "end": 2450.0, "text": " Because in the inference time"}, {"start": 2450.32, "end": 2455.04, "text": " You're obviously trying to generate something that does not exist. So you do not have any conditioning signal"}, {"start": 2455.36, "end": 2457.6, "text": " So that means you you won't have it here either"}, {"start": 2457.6, "end": 2463.52, "text": " So i'm not sure how they handle there is no explanation whatsoever of how this is exactly done in the paper"}, {"start": 2464.32, "end": 2467.44, "text": " What they might be doing is some form of um"}, {"start": 2467.84, "end": 2471.92, "text": " Just msc loss or something or simply unconditional generation"}, {"start": 2472.24, "end": 2475.68, "text": " And because during training you actually have the ground truth images"}, {"start": 2475.68, "end": 2481.52, "text": " You can use some form of like msc or something to just make sure that whatever is generated is closer"}, {"start": 2482.0, "end": 2483.52, "text": " to the output"}, {"start": 2483.52, "end": 2486.56, "text": " Again confusing that doesn't make complete sense"}, {"start": 2486.8799999999997, "end": 2492.8799999999997, "text": " I i'm still confused a bit by this if anyone has any any hypothesis feel free to just comment down below i'll be"}, {"start": 2493.2, "end": 2497.2799999999997, "text": " Very happy to to read through and and and discuss what's going on here"}, {"start": 2497.8399999999997, "end": 2499.3599999999997, "text": " In any case guys"}, {"start": 2499.3599999999997, "end": 2504.08, "text": " Uh, that's it. Let me again just show you quickly the the videos. They are fairly impressive"}, {"start": 2504.08, "end": 2507.04, "text": " Um, I I love this one with the with the doggo"}, {"start": 2507.2799999999997, "end": 2512.7999999999997, "text": " It's a dog wearing a superhero outfit with red cape flying through the sky and with that i'm gonna leave you here"}, {"start": 2513.04, "end": 2515.92, "text": " Hopefully you took out something from this video you learned something new"}, {"start": 2515.92, "end": 2534.7200000000003, "text": " If you did consider subscribing to this channel share the video out and until next time. Bye. Bye guys"}]