AudioCLIP: Extending CLIP to Image, Text and Audio | Paper Explained
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I cover AudioCLIP - OpenAI's CLIP (contrastive language-image pre-training) model extended to support the audio modality and novel query directions.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Paper: https://arxiv.org/abs/2106.13043
✅ Jina AI: https://github.com/jina-ai/jina
✅ My GitHub projects: https://github.com/gordicaleksa
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 

00:00 Overview of the CLIP model
07:15 ESResNeXt model and spectrograms
10:30 High-level system overview
11:25 AudioCLIP datasets walk-through
14:45 Audio data augmentations
16:30 Various training stages of AudioCLIP
19:10 Results
21:45 Querying results
23:00 Neural Search and Jina AI

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️


Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#audioclip #clip #neuralsearch
 What's cracking guys? In this video I'm covering audio clip, extending clip to image, text and audio by Andrei Guzov, Federico Rau, Yun Hees and Andreas Dengel from DFKI and Technical University of Kaiserlautern. So what they do, the main idea they do in this paper is they extend clip by adding a novel modality and that's audio. And so before even I start digging into this paper, I thought explaining how OpenAI's clip model works and for those of you who want to know all of the need to read the details of the paper, you can check out my video on clip. I've made it a couple of months ago and yeah. So having said that, the main motivation behind clip was to try and see how vision models perform given a huge amount of data, so scale. And as we already know, OpenAI did the same thing in NLP with GPT family of models, especially the GPT-3 model, which was huge, like 175 billion parameter model. And the thing is in vision, computer vision, until recently we were using stuff like ImageNet or ImageNet 21k, which were super small compared to the NLP standards. So obviously, okay, Google had their own proprietary big data set, JFT-300M, I think, and but that's not obviously public. And so OpenAI had to create their own data set and they finally ended up with 400 million image caption pairs, which they've used to train this clip model we're just gonna, we are about to explore. So again, one more thing is that they wanted to see how the natural language works as the source of supervision compared to those gold labels you have on your classical ImageNet data set, for example, or whatever arbitrary vision data set like Cypher 10 or whatnot. Okay, having said that, let me kind of briefly recap how clip works. So how you train clip is the following, you have a batch of images, you have a batch of corresponding captions, and you encode those into these representations here. And what you do then is you basically apply cosine similarity between all of the pairs. And you want to make sure you maximize the corresponding image and captions. So I1 corresponds to T1. That's why we want to maximize this particular term, but we want to minimize all of the other ones. And the same goes for the other elements on this main diagonal. And so again, you want to maximize those and you want to minimize all of the other ones. So going in a bit more detail here, ImageNet, so what they've used for the image encoder was the vision transformer. And so basically, you feed in an image and from a certain layer, you're just going to take a vector and you're going to linearly project that one. And that's how they obtain these encodings. On the other hand, text encoder was GPT-2. And how they formed this representation, so these T1 through Tn is the following. You input something called start of a sequence token, then you're going to encode this via byte pair encoding, for example. So you're going to embed those tokens here. And finally, you'll append the end of sentence token. And so the representation they finally use is after propagating this through the transformer, they'll just take the representation that's on top of EOS token. And they're just going to linearly project that in order to get these T1 through Tn, where n is the batch size. Then what they do is they L2 normalize these vectors so that they can just do dot product. And by doing dot product, you basically just get cosine similarity. So just to recap here, so A times B, so A dot product B is actually, so L2 norm of A times L2 norm of B times cosine of theta of the angle between those two vectors, okay? And since, as I said, these are normalized, that means these are is going to be one. So a dot product is the same thing as cosine similarity. So you're just gonna do element-wise multiplication between these embeddings, and you're going to end up with these. And finally, you're just gonna apply cross entropy, like symmetric cross entropy to this matrix, and that's going to be the last function of CLIP. So that will just push these terms to one, and it's gonna, so these ones will be pushed to one, and all of the other ones are going to be pushed down to minus one, okay? So all of the elements aside from the main diagonal are going to be pushed towards minus one. And that's in a nutshell how they train this multimodal embedding space. They back prop through the weights of the image encoder and the text encoder, and that's your CLIP. So a nice thing they later showed is that they can use this in a zero-shot setting on any target data set. So take, take for example, Cypher 10, I don't know, I think this is Cypher 10. So you have plane, car, dog, bird, whatever, and you want to convert that gold label into this format that CLIP is used to. And what they do is just they do simply something like this, a photo alpha, and like just insert like your label for sort of a photo alpha plane, I don't know. And you encode all of your labels into T1 through Tn, where n is the number of labels in your data set, so maybe like 10 for Cypher 10. And then how you classify an image, you just take your image, you encode it, and you do again simply like cosine similarity, you apply softmax, and you find the mode of the distribution, and that's your output. So here we have a photo of a dog. And so it's as simple as that you can kind of zero-shot create this classification head, and you can then classify your images. And so when I said classification head, let me just kind of break that down. If we take this I1 embedding, and I kind of draw it as a vector that has like maybe four dimensions, let's simplify it a bit. And so basically what these will do is they will just form the weights of the connections of that classifier head. So you'll have something like this, and this will be your first output, then you'll have, so this one, this embedding here, when you do multiplication with the I1 vector, you'll get something like this. So as you can see, you're just basically forming a classification head, and the reason we can do this is, as we saw here, is if we do dot product, so that's just element-wise multiplication, and that's pretty much the thing we're doing here with these connections, we do cosine similarity. And so as you can see, all of these blue, green, and other weights are formed, are pre-computed, and they form a specific classification head for this specific dataset at hand. So hopefully that was clear. Now that's CLIP, now let's jump into the paper. So they say here, in this work we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ES Resnext audio model into the CLIP framework using the audio set dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. The main unknown here, I guess, will be ES Resnext, which stands for, so ES stands for environmental sound. The X part is just a modification on top of the Resnet, the famous Resnet model with skip connections. So how those sound classification models work is pretty similar. So audio is simply a 1D signal, so you'll have something like this. You have your samples, 1D samples of your audio waveform, and that will look something like this. And you have two, pretty much two approaches to audio classification. You either apply something like 1D convolution, and then you process the signal until you get your classification output, or you convert this, so more commonly you just convert this into something called Spectrogram, where then you apply a 2D convolutional neural network to get your classification results. So Spectrogram just has, on the Y axis you have the frequency component, on the X axis you have the time component, and what the Spectrogram tells you is the following. So if I take a particular time point and a particular frequency, maybe like 1 kHz, it's going to tell you the amplitude of the sinusoid at that particular point. And why a sinusoid? Because if you know what Fourier transform is, so you can basically represent any natural signal as a sum of sinusoids with different frequencies, with different amplitudes and phases. So applying that Fourier transform, you can convert this arbitrary natural signal into Spectrogram. And that decomposition was just proven experimentally to be super useful, so that's why we are commonly using these Spectrograms, or male Spectrograms, which are just log-normalized Spectrograms as an input. And so, yes, ResNet is just a fancy, improved ResNet style architecture that works across these 2D Spectrograms. With a small caveat, they're not using this thing called short time Fourier transform, they're using something called complex frequency B-spline, super fancy, we don't even need to know the details, they just have some additional hyperparameters and what it turns out that this STFT, so this short time Fourier transform is just a special case of that thing, and so they can initialize it so that initially it behaves as STFT, but then eventually those parameters are learned and that showed to improve the results of this ES ResNext. So that's all the prerequisite knowledge you need, again, don't worry if you don't understand every single detail here, you just have some 2D Spectrogram and this ResNet model and that's how you, that's how this ES ResNext works like. Okay, now let's jump to the whole pipeline, this is how the final model looks like, and as you can see this is your standard clip, so this part here is your standard clip, and they additionally add, so this is the ES ResNext, you feed in audio, you find the representations, and as you can see here, this is like pretty much the same idea from clip, you just this time because you have the additional modality, you'll create these matrices of cosine similarities between audio and image, so this one between audio and text, that's this one, and again you have text and image, that's your classical clip, and that's this one, and you'll basically be again maximizing the main diagonal cosine similarities, you'll be suppressing the after the main diagonal elements, and that's how you're going to train this novel audio clip model. Okay, now let's see what else, so their procedure of training this thing is fairly complex, there are many different initializations, different training procedures, so let's briefly go through the data sets they are using, and they mentioned here, so in this work the clip data set was used indirectly as a weight initializer of the text and image heads, the clip model, so basically they implicitly used that huge open AIS data set by reusing their weights, that's what they want to say here, for the purposes of this work the ImageNet data set served as a weight initializer of the ES ResNext model, so in that previous paper that introduced this model ES ResNext, they showed that using ImageNet initialization helps boost the classification performance on sounds after they additionally train it on this thing called audio set, which we'll soon see. So aside from those two data sets which are used implicitly, they used this thing called audio set explicitly, so that's a data set that has 1.8 million, so it's a fairly big data set for the audio research community, and so it has 527 classes, whatnot, and each sample is a snippet up to 10 seconds long from a YouTube video defined by your corresponding ID and timings, but basically you usually ditch these components, so the vision component, you just focus on the clips and on the labels of those clips, but they say here, for this work we acquired video frames in addition to audio tracks, thus the audio set data set became the glue between the vanilla clip framework and our tri-modal extension on top of it. During the training part, 10 equally distant frames were extracted from our video recording, and one of them was picked randomly and passed through the audio clip model, so let's see how that works. So drawing a video here, something like this, I'm looking from it sideways, and these are frames, okay, and you'll have like spatial extent of the frame, whatnot, and that's not that important. Okay, so again, associated with this video, we'll have an audio, so we'll have like an audio here, so that will be some like natural signal, something like this, and again, we'll have a label, so I don't know, like cat or whatnot, and so how they perform, how they create a data set that will be used to jointly train all of the three modalities is the following. As they mentioned, they pick randomly one of these like frames, maybe this one, and so you have a 2D image there, so they are going to convert this into a 2D spectrogram, okay, so that's again a 2D image, and you have your label here, so those three will be fed into this system and used to jointly train it, and we'll see there are a bit more details than that, but like that's for start. They additionally have these two datasets called Urban Sound 8K and ESC50, which is Environment Sound Classification 50 dataset, and they use those in two ways, so they mentioned here on this dataset, we perform zero-shot inference using the audio clip model trained on audio set, so they use it to evaluate the models, but they also use those datasets to fine-tune the audio head, as we'll soon see. Aside from the datasets, they also use these data augmentations for the reason that's stated here. In comparison to the composite clip dataset, the audio datasets provide two orders of magnitude, less training samples, which makes overfitting an issue, so they have this thing called time scaling, where, so this is your like audio signal, and you have some like signal inside of it, and what they do is basically they either compress it, so you'll have a signal here, or they decompress it, so you'll have something like this, and they feed that into the ES Resnext model. They also use this thing called time inversion that's equivalent to flipping your image inside computer vision tasks, so that means, like, simply you're going to, if we have this signal here, and this is some like symmetry axis, you just basically rotate by 180 to get the signal, and then you feed that into ES Resnext. They also have random crops and padding, because they actually need to have these signals of appropriate length, so that they can be fed into the ES Resnext model, so sometimes they'll have to, if this is the, for example, if this is the target like length, that means they'll have to kind of pad this part here, they didn't say how they pad it, I guess it's zeros, or something, or if we have a longer signal, then you'll just crop it in order to fit into this length of the signal. So, and yeah, finally they have this random noise, they basically just have additive white gaussian noise applied, and that's similar to when you do photometric augmentations when you're training your computer vision models, so that's an equivalent type of augmentation. How they train this thing is the following. While the clip model was already pre-trained on text image pairs, we decided to perform an extended audio set pre-training of the audio head first, as it improved performance of the base ES Resnext model, and then to continue training in a tri-model setting, combining it with two other heads. Here, the whole audio clip model was trained jointly on the audio set data set, using audio snippets, the corresponding video frames, and the assigned textual labels. The whole training pipeline roughly consists out of three parts. So the first part is, you have clip, and it's already pre-trained by the OpenAI team, and so that's just taken as is. The second part is your ES Resnext, which is first initialized via these ImageNet weights, and then they additionally pre-train it using this audio set data set. The reason being is ImageNet weights are obviously more suited towards, like, understanding images, where they want to apply this model for audio, so that's why they have this second part here. Finally, and they mentioned it here, parameters of the two other sub-networks, namely text and image head, were frozen during the cooperative pre-training of the audio encoding head, thus these heads served as teachers in a multi-modal knowledge distillation setup. So that means the third component is, again, they're training on this audio set data set, but they this time do it jointly. So here they had a standalone training of ES Resnext, here they do it jointly. So when I say jointly, let me just go back here to the, to that drawing. Okay, so what I do is they freeze these, so they freeze these weights here, they freeze these weights here, and now they just back prop through this model, and they jointly train this whole system, but they just update the weights of the audio head. So that's the third part of how they train ES Resnext. So as I said, multiple, multiple details there, but yeah. And the last part is they actually train the whole system jointly, and they update all of the three models, as they mentioned here. Here all three modality dedicated heads were tuned together, making the resulting model take into account the distribution of images and textual descriptions, in addition to distribution of audio samples. So the reason they do this is because Clip was obviously trained on this OpenAI dataset, and here they want to evaluate this audio clip model on audio set, and so that's the reason why they additionally fine-tune all of the three heads onto audio set, because that kind of boosts the performance on the downstream tasks. And it makes sense, I guess. Okay, finally, let's see the results. First things first, they kind of show that the original pre-training on audio dataset of this ES Resnext model, so when they have like more epochs, that the performance gets better on the audio set, and it marginally gets better on these two other data sets. So that's kind of not that informative. Let's see some other tables they have here. So this is an important one. So they evaluate audio clip in two settings. First is audio head only, and then the full model. So audio head only is that when you only train the audio head while the other two, the text and the image heads, were frozen, and the full model is where you jointly train all of the three heads. And we can see on ImageNet, the performance actually goes down, which makes sense if you think about it, because the ES Resnext was initialized with ImageNet weights, and the more you fine-tune it jointly with the other heads, the more the model forgets those initial ImageNet weights, and the worse its performance gets on the ImageNet dataset. And again, hopefully you understand the setup here, so let me just kind of go back here. So what happens here is you have this setup, you take your ImageNet datasets, you'll have thousand labels, you'll encode them into these thousand embeddings, and then how you find the accuracy, you just take ImageNet images, and you feed them, and you apply this process to find the highest cosine similarity. That's how you're going to do the classification on ImageNet. Let's get back to the table. They also show results on audio set. Here we can have, we can classify images, we can also just embed audio using ES Resnext instead of image, and again, they show here that obviously the performance gets better, which makes sense since they are training jointly on this very same audio set dataset. They additionally report state-of-the-art results on this UrbanSound 8K and ESC50 datasets in the full model setup, which is cool. Not much new information here. They again have, like, these are the state-of-the-art results, and they report this zero shot. I kind of hate these tables, I love they had some lines here, but yeah. So this row here shows you the zero shot performance of the audio clip, and here in this row they actually pre-train on these datasets, and that's where they get the solar results, okay? The last thing where they report their results are on querying, and basically because we now have three modalities, that means you can query in multiple ways. You can query with an image, you can query audio, or you can query with text, you can query images, or via images, you can query audio. So all of the different directions are possible, and they test all of those, and again, let's see the results. So if you query with text for images on ImageNet, the performance actually drops if we go to the jointly trained model, which again makes sense, that's consistent with the previous results from the previous table, but again, on the audio set, they actually do improve on all of these different, like, combinations. So querying with text, where the result is an image, querying with text where the result is audio, they also query with audio where the result is image, and vice versa. And again, they have some small regression here on the ESC50, so basically the only dataset where they have improvements, but it's a big one, is this audio set, and all of the other results either regress or stay pretty much the same as prior to this joint training. So these querying capabilities are super important, because that's how we search for information. So if you think about Google, you basically have a textual query and retrieve bunch of different information that can be of various modalities. So it can be an image, it can be a video, it can be text, whatever. So Google used to be a rule-based system, and in the meanwhile, like a couple of years ago when BERT Transformer was published, Google started using BERT in production. So when you type something in Google, you have neural networks being used to retrieve certain results of your queries. And that's something called neural search. And when I'm added, let me just mention GINA-AI, which is an open source framework, which is super cool, because compared to Google, which is a general purpose framework, which you obviously cannot integrate into your own projects, you can use GINA-AI to kind of integrate this search functionality into your project, where they're using neural search. And as soon as models such as AudioClip are kind of published, GINA-AI developers integrate all of those novel research into their framework, and they make it easy to extend their framework with these novel models. So do check them out. I'm a huge fan of open source. If you've been following my channel for quite some time, you know that I'm bullish on open source. You can check out some of my previous projects on GitHub. If you're into that, I have a bunch of different deep learning projects implemented there from scratch. So do check those projects out. Do check GINA-AI. And until next time, subscribe, share out this video if you liked it, and bye bye.
[{"start": 0.0, "end": 5.92, "text": " What's cracking guys? In this video I'm covering audio clip, extending clip to image, text and audio"}, {"start": 5.92, "end": 14.56, "text": " by Andrei Guzov, Federico Rau, Yun Hees and Andreas Dengel from DFKI and Technical University of"}, {"start": 14.56, "end": 21.12, "text": " Kaiserlautern. So what they do, the main idea they do in this paper is they extend clip"}, {"start": 21.84, "end": 29.36, "text": " by adding a novel modality and that's audio. And so before even I start digging into this paper,"}, {"start": 29.36, "end": 34.96, "text": " I thought explaining how OpenAI's clip model works and for those of you who want to know all"}, {"start": 34.96, "end": 39.6, "text": " of the need to read the details of the paper, you can check out my video on clip. I've made it a"}, {"start": 39.6, "end": 47.92, "text": " couple of months ago and yeah. So having said that, the main motivation behind clip was to"}, {"start": 49.120000000000005, "end": 57.84, "text": " try and see how vision models perform given a huge amount of data, so scale. And as we already know,"}, {"start": 57.84, "end": 64.24000000000001, "text": " OpenAI did the same thing in NLP with GPT family of models, especially the GPT-3 model, which was"}, {"start": 64.24000000000001, "end": 72.48, "text": " huge, like 175 billion parameter model. And the thing is in vision, computer vision, until recently"}, {"start": 72.48, "end": 78.96000000000001, "text": " we were using stuff like ImageNet or ImageNet 21k, which were super small compared to the NLP"}, {"start": 78.96000000000001, "end": 86.56, "text": " standards. So obviously, okay, Google had their own proprietary big data set, JFT-300M, I think,"}, {"start": 86.56, "end": 93.44, "text": " and but that's not obviously public. And so OpenAI had to create their own data set and they"}, {"start": 93.44, "end": 99.84, "text": " finally ended up with 400 million image caption pairs, which they've used to train this clip model"}, {"start": 99.84, "end": 106.96000000000001, "text": " we're just gonna, we are about to explore. So again, one more thing is that they wanted to see"}, {"start": 106.96000000000001, "end": 112.72, "text": " how the natural language works as the source of supervision compared to those gold labels you have"}, {"start": 112.72, "end": 119.28, "text": " on your classical ImageNet data set, for example, or whatever arbitrary vision data set like"}, {"start": 119.28, "end": 125.92, "text": " Cypher 10 or whatnot. Okay, having said that, let me kind of briefly recap how clip works. So"}, {"start": 125.92, "end": 130.32, "text": " how you train clip is the following, you have a batch of images, you have a batch of corresponding"}, {"start": 131.2, "end": 138.24, "text": " captions, and you encode those into these representations here. And what you do then"}, {"start": 138.24, "end": 145.84, "text": " is you basically apply cosine similarity between all of the pairs. And you want to make sure you"}, {"start": 145.84, "end": 152.64000000000001, "text": " maximize the corresponding image and captions. So I1 corresponds to T1. That's why we want to maximize"}, {"start": 152.64000000000001, "end": 157.92000000000002, "text": " this particular term, but we want to minimize all of the other ones. And the same goes for the other"}, {"start": 157.92000000000002, "end": 164.88, "text": " elements on this main diagonal. And so again, you want to maximize those and you want to minimize"}, {"start": 164.88, "end": 171.35999999999999, "text": " all of the other ones. So going in a bit more detail here, ImageNet, so what they've used for"}, {"start": 171.35999999999999, "end": 178.48, "text": " the image encoder was the vision transformer. And so basically, you feed in an image and from a certain"}, {"start": 178.48, "end": 183.28, "text": " layer, you're just going to take a vector and you're going to linearly project that one. And"}, {"start": 183.28, "end": 192.16, "text": " that's how they obtain these encodings. On the other hand, text encoder was GPT-2. And how they"}, {"start": 192.16, "end": 199.68, "text": " formed this representation, so these T1 through Tn is the following. You input something called"}, {"start": 199.68, "end": 205.84, "text": " start of a sequence token, then you're going to encode this via byte pair encoding, for example."}, {"start": 205.84, "end": 213.68, "text": " So you're going to embed those tokens here. And finally, you'll append the end of sentence token."}, {"start": 213.68, "end": 219.2, "text": " And so the representation they finally use is after propagating this through the transformer,"}, {"start": 219.2, "end": 224.88, "text": " they'll just take the representation that's on top of EOS token. And they're just going to linearly"}, {"start": 224.88, "end": 231.83999999999997, "text": " project that in order to get these T1 through Tn, where n is the batch size. Then what they do is"}, {"start": 231.83999999999997, "end": 238.48, "text": " they L2 normalize these vectors so that they can just do dot product. And by doing dot product,"}, {"start": 238.48, "end": 245.67999999999998, "text": " you basically just get cosine similarity. So just to recap here, so A times B, so A dot product B"}, {"start": 245.68, "end": 254.32, "text": " is actually, so L2 norm of A times L2 norm of B times cosine of theta of the angle between those"}, {"start": 254.32, "end": 260.88, "text": " two vectors, okay? And since, as I said, these are normalized, that means these are is going to be one."}, {"start": 260.88, "end": 266.16, "text": " So a dot product is the same thing as cosine similarity. So you're just gonna do element-wise"}, {"start": 266.16, "end": 270.32, "text": " multiplication between these embeddings, and you're going to end up with these. And finally,"}, {"start": 270.32, "end": 276.4, "text": " you're just gonna apply cross entropy, like symmetric cross entropy to this matrix, and that's"}, {"start": 277.04, "end": 282.8, "text": " going to be the last function of CLIP. So that will just push these terms to one, and it's gonna,"}, {"start": 282.8, "end": 289.6, "text": " so these ones will be pushed to one, and all of the other ones are going to be pushed down to minus"}, {"start": 289.6, "end": 294.96, "text": " one, okay? So all of the elements aside from the main diagonal are going to be pushed towards minus"}, {"start": 294.96, "end": 300.71999999999997, "text": " one. And that's in a nutshell how they train this multimodal embedding space. They back prop through"}, {"start": 300.71999999999997, "end": 307.12, "text": " the weights of the image encoder and the text encoder, and that's your CLIP. So a nice thing they"}, {"start": 307.12, "end": 312.71999999999997, "text": " later showed is that they can use this in a zero-shot setting on any target data set. So take,"}, {"start": 312.71999999999997, "end": 317.76, "text": " take for example, Cypher 10, I don't know, I think this is Cypher 10. So you have plane, car, dog,"}, {"start": 317.76, "end": 325.44, "text": " bird, whatever, and you want to convert that gold label into this format that CLIP is used to. And"}, {"start": 325.44, "end": 330.48, "text": " what they do is just they do simply something like this, a photo alpha, and like just insert"}, {"start": 330.48, "end": 336.48, "text": " like your label for sort of a photo alpha plane, I don't know. And you encode all of your labels"}, {"start": 336.48, "end": 343.12, "text": " into T1 through Tn, where n is the number of labels in your data set, so maybe like 10 for Cypher 10."}, {"start": 343.12, "end": 349.44, "text": " And then how you classify an image, you just take your image, you encode it, and you do again simply"}, {"start": 350.24, "end": 355.84000000000003, "text": " like cosine similarity, you apply softmax, and you find the mode of the distribution, and that's your"}, {"start": 355.84000000000003, "end": 362.8, "text": " output. So here we have a photo of a dog. And so it's as simple as that you can kind of zero-shot"}, {"start": 362.8, "end": 367.76, "text": " create this classification head, and you can then classify your images. And so when I said"}, {"start": 367.76, "end": 375.84, "text": " classification head, let me just kind of break that down. If we take this I1 embedding, and I kind of"}, {"start": 376.96, "end": 384.15999999999997, "text": " draw it as a vector that has like maybe four dimensions, let's simplify it a bit. And so"}, {"start": 384.15999999999997, "end": 391.2, "text": " basically what these will do is they will just form the weights of the connections of that"}, {"start": 391.2, "end": 397.2, "text": " classifier head. So you'll have something like this, and this will be your first output, then you'll"}, {"start": 397.2, "end": 404.24, "text": " have, so this one, this embedding here, when you do multiplication with the I1 vector, you'll get"}, {"start": 404.24, "end": 408.64, "text": " something like this. So as you can see, you're just basically forming a classification head,"}, {"start": 408.64, "end": 415.28, "text": " and the reason we can do this is, as we saw here, is if we do dot product, so that's just element-wise"}, {"start": 415.28, "end": 419.84, "text": " multiplication, and that's pretty much the thing we're doing here with these connections, we do"}, {"start": 419.84, "end": 427.84, "text": " cosine similarity. And so as you can see, all of these blue, green, and other weights are formed,"}, {"start": 427.84, "end": 433.52, "text": " are pre-computed, and they form a specific classification head for this specific dataset"}, {"start": 433.52, "end": 438.88, "text": " at hand. So hopefully that was clear. Now that's CLIP, now let's jump into the paper. So they say"}, {"start": 438.88, "end": 444.23999999999995, "text": " here, in this work we present an extension of the CLIP model that handles audio in addition to text"}, {"start": 444.24, "end": 450.88, "text": " and images. Our proposed model incorporates the ES Resnext audio model into the CLIP framework"}, {"start": 450.88, "end": 456.24, "text": " using the audio set dataset. Such a combination enables the proposed model to perform bimodal"}, {"start": 456.24, "end": 461.04, "text": " and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen"}, {"start": 461.04, "end": 468.0, "text": " datasets in a zero-shot inference fashion. The main unknown here, I guess, will be ES Resnext,"}, {"start": 468.0, "end": 474.56, "text": " which stands for, so ES stands for environmental sound. The X part is just a modification on top"}, {"start": 474.56, "end": 479.6, "text": " of the Resnet, the famous Resnet model with skip connections. So how those sound classification"}, {"start": 479.6, "end": 485.2, "text": " models work is pretty similar. So audio is simply a 1D signal, so you'll have something like this."}, {"start": 485.2, "end": 492.56, "text": " You have your samples, 1D samples of your audio waveform, and that will look something like this."}, {"start": 492.56, "end": 499.04, "text": " And you have two, pretty much two approaches to audio classification. You either apply something"}, {"start": 499.04, "end": 505.92, "text": " like 1D convolution, and then you process the signal until you get your classification output,"}, {"start": 505.92, "end": 512.24, "text": " or you convert this, so more commonly you just convert this into something called Spectrogram,"}, {"start": 512.24, "end": 517.84, "text": " where then you apply a 2D convolutional neural network to get your classification results."}, {"start": 517.84, "end": 523.84, "text": " So Spectrogram just has, on the Y axis you have the frequency component, on the X axis you have"}, {"start": 523.84, "end": 529.36, "text": " the time component, and what the Spectrogram tells you is the following. So if I take a particular"}, {"start": 529.36, "end": 536.32, "text": " time point and a particular frequency, maybe like 1 kHz, it's going to tell you the amplitude"}, {"start": 536.88, "end": 545.2, "text": " of the sinusoid at that particular point. And why a sinusoid? Because if you know what Fourier"}, {"start": 545.2, "end": 551.9200000000001, "text": " transform is, so you can basically represent any natural signal as a sum of sinusoids with"}, {"start": 551.9200000000001, "end": 557.0400000000001, "text": " different frequencies, with different amplitudes and phases. So applying that Fourier transform,"}, {"start": 557.0400000000001, "end": 563.5200000000001, "text": " you can convert this arbitrary natural signal into Spectrogram. And that decomposition was just"}, {"start": 563.5200000000001, "end": 569.6, "text": " proven experimentally to be super useful, so that's why we are commonly using these Spectrograms,"}, {"start": 569.6, "end": 576.8000000000001, "text": " or male Spectrograms, which are just log-normalized Spectrograms as an input. And so, yes, ResNet is"}, {"start": 576.8000000000001, "end": 584.5600000000001, "text": " just a fancy, improved ResNet style architecture that works across these 2D Spectrograms."}, {"start": 585.2, "end": 589.76, "text": " With a small caveat, they're not using this thing called short time Fourier transform,"}, {"start": 589.76, "end": 595.84, "text": " they're using something called complex frequency B-spline, super fancy, we don't even need to know"}, {"start": 595.84, "end": 600.72, "text": " the details, they just have some additional hyperparameters and what it turns out that this"}, {"start": 600.72, "end": 606.72, "text": " STFT, so this short time Fourier transform is just a special case of that thing, and so they can"}, {"start": 606.72, "end": 612.4, "text": " initialize it so that initially it behaves as STFT, but then eventually those parameters are"}, {"start": 612.4, "end": 617.9200000000001, "text": " learned and that showed to improve the results of this ES ResNext. So that's all the prerequisite"}, {"start": 617.9200000000001, "end": 622.0, "text": " knowledge you need, again, don't worry if you don't understand every single detail here,"}, {"start": 622.0, "end": 626.96, "text": " you just have some 2D Spectrogram and this ResNet model and that's how you, that's how this ES"}, {"start": 626.96, "end": 634.24, "text": " ResNext works like. Okay, now let's jump to the whole pipeline, this is how the final model looks"}, {"start": 634.24, "end": 638.88, "text": " like, and as you can see this is your standard clip, so this part here is your standard clip,"}, {"start": 639.44, "end": 644.48, "text": " and they additionally add, so this is the ES ResNext, you feed in audio, you find the"}, {"start": 644.48, "end": 651.04, "text": " representations, and as you can see here, this is like pretty much the same idea from clip, you just"}, {"start": 651.04, "end": 655.68, "text": " this time because you have the additional modality, you'll create these matrices of"}, {"start": 655.68, "end": 662.88, "text": " cosine similarities between audio and image, so this one between audio and text, that's this one,"}, {"start": 662.88, "end": 668.3199999999999, "text": " and again you have text and image, that's your classical clip, and that's this one, and you'll"}, {"start": 668.3199999999999, "end": 674.4, "text": " basically be again maximizing the main diagonal cosine similarities, you'll be suppressing the"}, {"start": 674.4, "end": 679.5999999999999, "text": " after the main diagonal elements, and that's how you're going to train this novel audio clip model."}, {"start": 679.6, "end": 688.5600000000001, "text": " Okay, now let's see what else, so their procedure of training this thing is fairly complex, there are"}, {"start": 688.5600000000001, "end": 694.5600000000001, "text": " many different initializations, different training procedures, so let's briefly go through the data"}, {"start": 694.5600000000001, "end": 700.08, "text": " sets they are using, and they mentioned here, so in this work the clip data set was used"}, {"start": 700.08, "end": 706.5600000000001, "text": " indirectly as a weight initializer of the text and image heads, the clip model, so basically they"}, {"start": 706.56, "end": 712.16, "text": " implicitly used that huge open AIS data set by reusing their weights, that's what they want to"}, {"start": 712.16, "end": 718.0799999999999, "text": " say here, for the purposes of this work the ImageNet data set served as a weight initializer of the"}, {"start": 718.0799999999999, "end": 725.04, "text": " ES ResNext model, so in that previous paper that introduced this model ES ResNext, they showed that"}, {"start": 725.04, "end": 732.4799999999999, "text": " using ImageNet initialization helps boost the classification performance on sounds after they"}, {"start": 732.48, "end": 737.28, "text": " additionally train it on this thing called audio set, which we'll soon see. So aside from those two"}, {"start": 737.28, "end": 743.12, "text": " data sets which are used implicitly, they used this thing called audio set explicitly, so that's"}, {"start": 743.12, "end": 750.4, "text": " a data set that has 1.8 million, so it's a fairly big data set for the audio research community,"}, {"start": 750.4, "end": 758.0, "text": " and so it has 527 classes, whatnot, and each sample is a snippet up to 10 seconds long from"}, {"start": 758.0, "end": 762.48, "text": " a YouTube video defined by your corresponding ID and timings, but basically you usually ditch these"}, {"start": 763.04, "end": 767.68, "text": " components, so the vision component, you just focus on the clips and on the labels of those clips,"}, {"start": 768.32, "end": 775.84, "text": " but they say here, for this work we acquired video frames in addition to audio tracks, thus the audio"}, {"start": 775.84, "end": 782.48, "text": " set data set became the glue between the vanilla clip framework and our tri-modal extension on top"}, {"start": 782.48, "end": 787.36, "text": " of it. During the training part, 10 equally distant frames were extracted from our video recording,"}, {"start": 787.36, "end": 793.2, "text": " and one of them was picked randomly and passed through the audio clip model, so let's see how"}, {"start": 793.2, "end": 799.84, "text": " that works. So drawing a video here, something like this, I'm looking from it sideways, and these are"}, {"start": 799.84, "end": 806.8000000000001, "text": " frames, okay, and you'll have like spatial extent of the frame, whatnot, and that's not that important."}, {"start": 806.8000000000001, "end": 813.9200000000001, "text": " Okay, so again, associated with this video, we'll have an audio, so we'll have like an audio here,"}, {"start": 813.92, "end": 820.4, "text": " so that will be some like natural signal, something like this, and again, we'll have a label, so I"}, {"start": 820.4, "end": 827.04, "text": " don't know, like cat or whatnot, and so how they perform, how they create a data set that will be"}, {"start": 827.04, "end": 831.8399999999999, "text": " used to jointly train all of the three modalities is the following. As they mentioned, they pick"}, {"start": 831.8399999999999, "end": 838.8, "text": " randomly one of these like frames, maybe this one, and so you have a 2D image there, so they are going"}, {"start": 838.8, "end": 844.7199999999999, "text": " to convert this into a 2D spectrogram, okay, so that's again a 2D image, and you have your label"}, {"start": 844.7199999999999, "end": 854.16, "text": " here, so those three will be fed into this system and used to jointly train it, and we'll see there"}, {"start": 854.16, "end": 859.04, "text": " are a bit more details than that, but like that's for start. They additionally have these two"}, {"start": 859.04, "end": 866.7199999999999, "text": " datasets called Urban Sound 8K and ESC50, which is Environment Sound Classification 50 dataset,"}, {"start": 866.72, "end": 871.6, "text": " and they use those in two ways, so they mentioned here on this dataset, we perform zero-shot"}, {"start": 871.6, "end": 878.48, "text": " inference using the audio clip model trained on audio set, so they use it to evaluate the models,"}, {"start": 878.48, "end": 885.6800000000001, "text": " but they also use those datasets to fine-tune the audio head, as we'll soon see. Aside from the"}, {"start": 885.6800000000001, "end": 890.88, "text": " datasets, they also use these data augmentations for the reason that's stated here. In comparison"}, {"start": 890.88, "end": 897.68, "text": " to the composite clip dataset, the audio datasets provide two orders of magnitude, less training"}, {"start": 897.68, "end": 903.36, "text": " samples, which makes overfitting an issue, so they have this thing called time scaling, where, so this"}, {"start": 903.36, "end": 910.24, "text": " is your like audio signal, and you have some like signal inside of it, and what they do is basically"}, {"start": 910.24, "end": 915.04, "text": " they either compress it, so you'll have a signal here, or they decompress it, so you'll have something"}, {"start": 915.04, "end": 921.68, "text": " like this, and they feed that into the ES Resnext model. They also use this thing called time"}, {"start": 921.68, "end": 928.16, "text": " inversion that's equivalent to flipping your image inside computer vision tasks, so that means, like,"}, {"start": 928.16, "end": 936.24, "text": " simply you're going to, if we have this signal here, and this is some like symmetry axis,"}, {"start": 936.24, "end": 943.04, "text": " you just basically rotate by 180 to get the signal, and then you feed that into ES Resnext."}, {"start": 943.04, "end": 950.24, "text": " They also have random crops and padding, because they actually need to have these signals of"}, {"start": 950.24, "end": 955.76, "text": " appropriate length, so that they can be fed into the ES Resnext model, so sometimes they'll have to,"}, {"start": 955.76, "end": 963.12, "text": " if this is the, for example, if this is the target like length, that means they'll have to kind of"}, {"start": 963.12, "end": 969.04, "text": " pad this part here, they didn't say how they pad it, I guess it's zeros, or something, or if we have"}, {"start": 969.04, "end": 975.4399999999999, "text": " a longer signal, then you'll just crop it in order to fit into this length of the signal."}, {"start": 975.4399999999999, "end": 981.8399999999999, "text": " So, and yeah, finally they have this random noise, they basically just have additive white gaussian"}, {"start": 981.8399999999999, "end": 987.5999999999999, "text": " noise applied, and that's similar to when you do photometric augmentations when you're training"}, {"start": 987.5999999999999, "end": 994.64, "text": " your computer vision models, so that's an equivalent type of augmentation. How they train this thing is"}, {"start": 994.64, "end": 1000.3199999999999, "text": " the following. While the clip model was already pre-trained on text image pairs, we decided to"}, {"start": 1000.3199999999999, "end": 1005.76, "text": " perform an extended audio set pre-training of the audio head first, as it improved performance of"}, {"start": 1005.76, "end": 1011.4399999999999, "text": " the base ES Resnext model, and then to continue training in a tri-model setting, combining it"}, {"start": 1011.4399999999999, "end": 1016.24, "text": " with two other heads. Here, the whole audio clip model was trained jointly on the audio set data"}, {"start": 1016.24, "end": 1021.04, "text": " set, using audio snippets, the corresponding video frames, and the assigned textual labels."}, {"start": 1021.04, "end": 1026.8799999999999, "text": " The whole training pipeline roughly consists out of three parts. So the first part is, you have clip,"}, {"start": 1026.8799999999999, "end": 1033.04, "text": " and it's already pre-trained by the OpenAI team, and so that's just taken as is. The second part is"}, {"start": 1033.04, "end": 1039.68, "text": " your ES Resnext, which is first initialized via these ImageNet weights, and then they additionally"}, {"start": 1039.68, "end": 1045.36, "text": " pre-train it using this audio set data set. The reason being is ImageNet weights are obviously"}, {"start": 1045.36, "end": 1050.8799999999999, "text": " more suited towards, like, understanding images, where they want to apply this model for audio,"}, {"start": 1050.88, "end": 1055.2800000000002, "text": " so that's why they have this second part here. Finally, and they mentioned it here,"}, {"start": 1056.0800000000002, "end": 1061.8400000000001, "text": " parameters of the two other sub-networks, namely text and image head, were frozen during the"}, {"start": 1061.8400000000001, "end": 1066.88, "text": " cooperative pre-training of the audio encoding head, thus these heads served as teachers in a"}, {"start": 1066.88, "end": 1072.4, "text": " multi-modal knowledge distillation setup. So that means the third component is, again, they're"}, {"start": 1072.4, "end": 1079.2, "text": " training on this audio set data set, but they this time do it jointly. So here they had a standalone"}, {"start": 1079.2, "end": 1084.0800000000002, "text": " training of ES Resnext, here they do it jointly. So when I say jointly, let me just go back here"}, {"start": 1084.0800000000002, "end": 1090.96, "text": " to the, to that drawing. Okay, so what I do is they freeze these, so they freeze these weights here,"}, {"start": 1090.96, "end": 1097.3600000000001, "text": " they freeze these weights here, and now they just back prop through this model, and they jointly"}, {"start": 1097.3600000000001, "end": 1102.4, "text": " train this whole system, but they just update the weights of the audio head. So that's the third"}, {"start": 1102.4, "end": 1109.1200000000001, "text": " part of how they train ES Resnext. So as I said, multiple, multiple details there, but yeah. And"}, {"start": 1109.1200000000001, "end": 1113.92, "text": " the last part is they actually train the whole system jointly, and they update all of the three"}, {"start": 1113.92, "end": 1119.3600000000001, "text": " models, as they mentioned here. Here all three modality dedicated heads were tuned together,"}, {"start": 1119.3600000000001, "end": 1124.48, "text": " making the resulting model take into account the distribution of images and textual descriptions,"}, {"start": 1124.48, "end": 1130.16, "text": " in addition to distribution of audio samples. So the reason they do this is because Clip was"}, {"start": 1130.16, "end": 1137.2, "text": " obviously trained on this OpenAI dataset, and here they want to evaluate this audio clip model on"}, {"start": 1137.2, "end": 1143.0400000000002, "text": " audio set, and so that's the reason why they additionally fine-tune all of the three heads"}, {"start": 1143.0400000000002, "end": 1149.1200000000001, "text": " onto audio set, because that kind of boosts the performance on the downstream tasks. And"}, {"start": 1150.5600000000002, "end": 1156.88, "text": " it makes sense, I guess. Okay, finally, let's see the results. First things first, they kind of show"}, {"start": 1156.88, "end": 1164.72, "text": " that the original pre-training on audio dataset of this ES Resnext model, so when they have like"}, {"start": 1164.72, "end": 1170.88, "text": " more epochs, that the performance gets better on the audio set, and it marginally gets better on"}, {"start": 1170.88, "end": 1177.44, "text": " these two other data sets. So that's kind of not that informative. Let's see some other tables they"}, {"start": 1177.44, "end": 1183.1200000000001, "text": " have here. So this is an important one. So they evaluate audio clip in two settings. First is audio"}, {"start": 1183.12, "end": 1188.56, "text": " head only, and then the full model. So audio head only is that when you only train the audio head"}, {"start": 1188.56, "end": 1193.76, "text": " while the other two, the text and the image heads, were frozen, and the full model is where you"}, {"start": 1193.76, "end": 1198.9599999999998, "text": " jointly train all of the three heads. And we can see on ImageNet, the performance actually goes down,"}, {"start": 1198.9599999999998, "end": 1205.52, "text": " which makes sense if you think about it, because the ES Resnext was initialized with ImageNet"}, {"start": 1205.52, "end": 1212.2399999999998, "text": " weights, and the more you fine-tune it jointly with the other heads, the more the model forgets"}, {"start": 1212.24, "end": 1219.44, "text": " those initial ImageNet weights, and the worse its performance gets on the ImageNet dataset. And again,"}, {"start": 1219.44, "end": 1225.1200000000001, "text": " hopefully you understand the setup here, so let me just kind of go back here. So what happens here"}, {"start": 1225.1200000000001, "end": 1230.32, "text": " is you have this setup, you take your ImageNet datasets, you'll have thousand labels, you'll"}, {"start": 1230.32, "end": 1236.0, "text": " encode them into these thousand embeddings, and then how you find the accuracy, you just take"}, {"start": 1236.0, "end": 1241.6, "text": " ImageNet images, and you feed them, and you apply this process to find the highest cosine similarity."}, {"start": 1241.6, "end": 1247.9199999999998, "text": " That's how you're going to do the classification on ImageNet. Let's get back to the table."}, {"start": 1248.48, "end": 1254.8799999999999, "text": " They also show results on audio set. Here we can have, we can classify images, we can also just embed"}, {"start": 1254.8799999999999, "end": 1262.48, "text": " audio using ES Resnext instead of image, and again, they show here that obviously the performance"}, {"start": 1262.48, "end": 1267.04, "text": " gets better, which makes sense since they are training jointly on this very same audio set"}, {"start": 1267.04, "end": 1274.48, "text": " dataset. They additionally report state-of-the-art results on this UrbanSound 8K and ESC50 datasets"}, {"start": 1274.48, "end": 1282.24, "text": " in the full model setup, which is cool. Not much new information here. They again have, like,"}, {"start": 1282.24, "end": 1287.68, "text": " these are the state-of-the-art results, and they report this zero shot. I kind of hate these tables,"}, {"start": 1288.32, "end": 1294.24, "text": " I love they had some lines here, but yeah. So this row here shows you the zero shot performance of"}, {"start": 1294.24, "end": 1300.32, "text": " the audio clip, and here in this row they actually pre-train on these datasets, and that's where they"}, {"start": 1300.32, "end": 1306.32, "text": " get the solar results, okay? The last thing where they report their results are on querying, and"}, {"start": 1306.32, "end": 1310.64, "text": " basically because we now have three modalities, that means you can query in multiple ways. You can"}, {"start": 1310.64, "end": 1316.8, "text": " query with an image, you can query audio, or you can query with text, you can query images, or"}, {"start": 1316.8, "end": 1322.48, "text": " via images, you can query audio. So all of the different directions are possible, and they test"}, {"start": 1322.48, "end": 1330.4, "text": " all of those, and again, let's see the results. So if you query with text for images on ImageNet,"}, {"start": 1330.4, "end": 1336.0, "text": " the performance actually drops if we go to the jointly trained model, which again makes sense,"}, {"start": 1336.72, "end": 1342.16, "text": " that's consistent with the previous results from the previous table, but again, on the audio set,"}, {"start": 1342.16, "end": 1349.52, "text": " they actually do improve on all of these different, like, combinations. So querying with text,"}, {"start": 1349.52, "end": 1354.48, "text": " where the result is an image, querying with text where the result is audio, they also query with"}, {"start": 1354.48, "end": 1360.72, "text": " audio where the result is image, and vice versa. And again, they have some small regression here"}, {"start": 1360.72, "end": 1367.68, "text": " on the ESC50, so basically the only dataset where they have improvements, but it's a big one, is"}, {"start": 1367.68, "end": 1375.2, "text": " this audio set, and all of the other results either regress or stay pretty much the same as prior to"}, {"start": 1375.2, "end": 1380.0800000000002, "text": " this joint training. So these querying capabilities are super important, because that's how we search"}, {"start": 1380.0800000000002, "end": 1385.28, "text": " for information. So if you think about Google, you basically have a textual query and retrieve"}, {"start": 1385.28, "end": 1389.52, "text": " bunch of different information that can be of various modalities. So it can be an image,"}, {"start": 1389.52, "end": 1394.32, "text": " it can be a video, it can be text, whatever. So Google used to be a rule-based system,"}, {"start": 1394.32, "end": 1398.96, "text": " and in the meanwhile, like a couple of years ago when BERT Transformer was published,"}, {"start": 1398.96, "end": 1404.72, "text": " Google started using BERT in production. So when you type something in Google, you have neural"}, {"start": 1404.72, "end": 1410.96, "text": " networks being used to retrieve certain results of your queries. And that's something called neural"}, {"start": 1410.96, "end": 1416.96, "text": " search. And when I'm added, let me just mention GINA-AI, which is an open source framework,"}, {"start": 1416.96, "end": 1421.6000000000001, "text": " which is super cool, because compared to Google, which is a general purpose framework, which you"}, {"start": 1421.6, "end": 1429.36, "text": " obviously cannot integrate into your own projects, you can use GINA-AI to kind of integrate this"}, {"start": 1429.36, "end": 1435.6, "text": " search functionality into your project, where they're using neural search. And as soon as models"}, {"start": 1435.6, "end": 1441.6, "text": " such as AudioClip are kind of published, GINA-AI developers integrate all of those novel research"}, {"start": 1441.6, "end": 1448.24, "text": " into their framework, and they make it easy to extend their framework with these novel models."}, {"start": 1448.24, "end": 1453.76, "text": " So do check them out. I'm a huge fan of open source. If you've been following my channel for"}, {"start": 1453.76, "end": 1457.92, "text": " quite some time, you know that I'm bullish on open source. You can check out some of my previous"}, {"start": 1457.92, "end": 1463.36, "text": " projects on GitHub. If you're into that, I have a bunch of different deep learning projects"}, {"start": 1463.36, "end": 1469.04, "text": " implemented there from scratch. So do check those projects out. Do check GINA-AI. And until next"}, {"start": 1469.04, "end": 1483.76, "text": " time, subscribe, share out this video if you liked it, and bye bye."}]