Neural Search with Jina AI | Open-source ML Tool Explained
👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany

In this video I kick off a series of videos where I'll be walking you through how to use certain ML tools!

ML is definitely more than research. It's about engineering and it's about using the best tools out there to get the job done. 

First up: Jina AI - a neural search engine!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ Jina AI GitHub: https://github.com/jina-ai/jina
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
00:00 A high-level overview of Jina AI
03:30 Setting up the MNIST fashion example
04:50 Arguments and data loading
10:15 Core concepts - Flow and Executors
11:40 Visualizing the flow
12:40 Flow is lazy
13:45 The core algorithm explained
15:20 Indexing
18:15 Encoding the images via SVD
22:45 Evaluation - finding closest embeddings
27:50 Writing to an HTML
29:25 HTML results visualized
30:10 Chatbot example overview
31:00 Outro

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ 

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon! 

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donation: 
https://www.paypal.com/paypalme/theaiepiphany 

Much love! ❤️

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
Bartłomiej Danek
Zvonimir Sabljic

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR ML PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#jinaai #neuralsearch #mltools
 What's cracking guys in this video I'm continuing with the coding video series and I'm not going to cover a research paper like code in this one I'm gonna cover open source ML tools and libraries and hopefully this will help you improve your own projects by including these awesome open source libraries and tools so in this one I'm gonna cover Gini AI and as you can see here they are a cloud native neural search framework for any kind of data so this enables it to import lightweight Google into your project whereas the neural part means you'll be using deep neural networks to perform the search and the any kind of data means basically you can take whatever modality you wish so like image or text or audio or video or mesh or whatnot and you can use those to query and find like data from any other modalities you can use images to query for certain text snippets you can use text to query for images or videos or audio for mesh or whatever like whatever combination in that combinatorial space you can imagine Gini AI can kind of help you do that as you can see the readme is pretty nicely structured they have lots of resources here to tutorials they have like lots of contributors people are using this already it's an established tool as you can see here used by 130 like organizations and people already and so the way I suggest you go ahead and explore this tool is by like understanding these quick demos and in this video I'm gonna walk you through step-by-step through the fashion image search aside from that you should probably go and understand the basic concepts of Gina so that's document executor and flow and they have really nice pages where they explain how you can use the document how you can use all of these like a native concepts to Gina so document is an abstraction they use to describe whatever data type you have so basically document can be an image document can be a text document can be video or whatever data type you have don't get confused by the terminology so document if you're like me a document is gonna associate you to text data textual data only and even the emoji here is associating us to to text but it's not it's a it's an abstraction object which wraps up any kind of data then they have executors which process the data and flows finally organize those executors into these kinds of pipelines as you can see here on the image a short note on the installation procedure because they are cloud native neural search engine that basically means because cloud is mostly Linux that means they support Gina for Linux pretty much and because I'm on Windows this thing does not work on Windows and I had to install a window subsystem for Linux I had some problems getting into work but after you have upgraded to WSL 2 and fix some other small problems like I had problems plotting because Matplotlib would not display the actual like chart so I had to do some tweaks so if you know you want to know all of those details if you are on Windows like me and you want to try this thing ago do join the discord community because I'm gonna gonna paste like the step-by-step how I kind of solved all of the problems I encountered on Windows and how I got it to work so having said that let's start with the fashion image search so they have a nice page here explaining how you can actually get it like running and the the end like result will be this you can see here basically we have a query image on the left and we can fetch like they are fetching 50 closest images from the data set where closeness is defined as you can see here like visual similarity in the case of images so yeah let's get this project started okay so the first thing you obviously need to do here is do git clone clone the repository on your own machine install the condo environment and just do pip install you Gina and that's gonna install Gina AI package to your environment and then you can get started so as you can see here I'm Windows so I'm in this like WSL terminal I'm using Ubuntu as my distribution of choice I'm just going to start Visual Studio code you can use whatever editor you prefer just like kind of navigate to the root directory of your project and again from your environment where you're using conda or something else just start your ID and then we can get started the first thing we will want to do is to clone the demo example to the root directory so I'm just gonna type Gina hello fork fashion into Gina fashion demo directory so basically this means hey I want to use Gina and I want to use these quick examples I want to fork it to my local machine I want to use fashion example in particular and I'm gonna fork it into Gina fashion demo so if I run that the as you can see here I just got this new folder downloaded and I'm gonna open up the the app.py so this is the main like main file and the thing we're going to do so we basically don't need any additional arguments so we can just start this thing and that's it so I'm gonna run f5 and start this project okay there are some basic parameters we are setting up here let me quickly just walk you through them so basically you can see here what is important is this part here so what they include they have this working directory the work directory is just going to create a random directory inside of your root directory and that's pretty much the only thing we care about in this one and then what they do is they as you can see here they have the data URL the the labels like for the training set so these two are for the training set so we are downloading fashion MNIST here and then they have the query data so basically the let's call these these are test sets again fashion MNIST and we have the data and we have the the labels here so number queries is just the number of queries that we're gonna display on the on the web page we'll see soon see and finally top k means how many how many similar items are we trying to find for each of these queries so those are all of the parameters you should care about and now let's return to the app.py so let's kind of step into this function and see what's going on so first things first they are going to create this random directory so that's it and then they're gonna form this this basically dictionary where they have as you can see here so the URL for the data and the local file name name where this data is gonna be saved so we're gonna save the the the training data directly inside of this random directory and we're gonna name this index labels file so nothing special here we're gonna do that for both the training set and the test set and that's it okay let's continue so next step is downloading the data so let's step into this download data function so basically nothing smart here this is everything is handled by this URL lib package so what happens is we're gonna skip this because it's none so we're gonna install this opener whatever so this is just like boilerplate pretty much you don't need to understand all of these details with progress bar just a context manager to show the progress while the thing is being downloaded and now we're gonna iterate through the dictionary and download the data and the labels for the training set and for the test set that's it so if it if it doesn't exist and it doesn't exist because we just created this random folder we're gonna retrieve it here we're gonna download it so until this point we just downloaded the data into those files and now we're gonna actually load the data from the files let's see load labels let's step into that one and it's very simple what they do is just they just open up that like local file and let me show you where it is so basically this is the random directory I was mentioning so it's directly inside of Gina root you open it up and you can see index label so that's the first file so now here we're gonna open it up as you can see here and we're gonna load it from using a numpy and just reading the file here so nothing you need to care about so this is some specific knowledge because they know the structure of this file they know that they have to offset reading eight bytes from the very start and that's it then they do the reshape part and this is always useful to have shapes in your mind constantly when you're debugging machine learning code so this should be a 60 K comma one should be the shape of this of this object because those are labels and we have 60 K examples inside of the index data set so I'm gonna step over here and we can check the data to see whether we are correct here so V I'm gonna find the data array and you can see here shape is 60 K comma one so that's it now we're gonna do the same thing for for the next label so this is gonna download the labels for the query data set same procedure so I won't be walking you through the same function again and finally we have we're downloading the actual images through that's done you can see in the directory we have this index originals that means these are actually the images okay and now we're gonna step into this load MNIST I'm gonna step into it and you can see again we just open up the file we read it we like have this non pipe from buffer and we reshape it into 60 K 28 by 28 because we have MNIST images here so that's it it's fairly simple I'm gonna just step over this and we're gonna see that after I do this so we data and we have 60 K 28 28 those are obviously MNIST images and finally last step we're gonna do the same thing so we're doing the same thing just for the query data set this time and I'm gonna step over this time so we're gonna load the data and that's it we downloaded the data we loaded the labels we loaded the like images and that's it so that data is currently stored into this dictionary so we have targets you can pick whatever type of data you want so for example we can we can pick index labels so that means the index data set the labels and you'll find the data array here so it's 60 K 1 you can do the same thing for example index which is images and you can see the data 60 K 28 28 so everything is stored into this neatly stored into this dictionary object and that's it now this is some nothing that important just some optimization stuff and finally this is the flow object I mentioned in the initial part of the video which is one of the like building blocks of these GNA like programs and what flow does is just organizes these executors into this nice pipeline which you can then run and execute a process the data in whichever whatever manner you wish so as you can see here we're not form flow and we're gonna add these so these are called executors so we have executor that's called my encoder we're gonna see how that thing what the thing does in a couple of minutes we have we'll have two of those in parallel then we'll have the indexer and the indexer is gonna index this thing into this rug directory which is the random UUID directory as you can see here the string on the screen and finally we have this my evaluator executor which is going to do some the actual searching and matching of the image embedding vector of the query embedding vector to the all of the other index embedding vectors so that's that's it and now because this is lazy execution this object still the constructors of these executors are still not called that's gonna happen after we open it up with a context manager here so only here will they actually execute and form those those like executors so that's the lazy execution design pattern so the thing I'm gonna do now is actually visualize this flow so I'm gonna stop this program I'm gonna add the plotting function here so we can plot you can actually plot these like flows and we're gonna just just save it into like I don't know like time flow SVG and that's it I'm gonna save this I'm gonna rerun it and I'm gonna show you what happens so we are here if I do a step over let's just step over it's gonna form this SVG and I'm gonna open it up and show you how this thing looks like here it is it's you can see the temporary flow SVG file in the root directory and for some reason it's gonna open it up in Microsoft Edge even though my default browser is Google Chrome so let's open it up here and you can see the actual like diagram so we have the input here we split the input into two my encoder executors then we have the my indexer and then we have the my evaluator if we were to put some names as the arguments to those here basically name they have a name parameter blah blah blah we have nicer diagrams but you get the point okay having done that let me now show you what I meant by delays execution so we're here now I still haven't executed this line so if I go to let me find the files let me find the the actual my executors folder so I'm gonna put some breakpoint inside of the constructors here so we need functions my indexer I'm gonna put a breaking point here my encoder and finally I'm gonna put a breaking point into my evaluator and so now I'm gonna put a breakpoint here and I'm gonna step over this line here and we're gonna see it's gonna hit the constructors of those executors so f10 and you can see here my encoder is hit here so I'm gonna do f5 again we we had encoder again because we have is remember two encoder executors in parallel so I'm gonna step over f10 so f5 indexer and f5 evaluator and that's it f5 will bring us to the next line f index okay so you saw that by doing this we only then executed the constructors of all of the executors you've inputted here and now now we're gonna hit the index function so first let me break this down on a high level for you and then we're gonna step over the code so we have as you can see here the main function there is not much lines of code here left it's fairly simple we have we're calling flow index which is the same thing as if if we were to call this post function and instead of using slash eval here we'd be using slash index so these are the same things so the indexing function is gonna as you can imagine kind of index the data and gonna create the embedding vectors and then the eval function is gonna do the actual matching so once we call the eval function we're gonna actually find the 50 closest matches to that query image okay so that's like the rough pipeline and then finally after we have the data we're gonna write it into HTML and we're gonna be able to open up this HTML page in the browser and see the final results let's go a bit deeper and then I'm gonna step into these this code so next thing you see we have these generators we have index generator here we have query generator here and we have some additional parameters not that important show progress is just gonna show us in the bar as the testing is being indexed and evaluated we're gonna see the progress on the in the console print results so once this thing is done the print result is a callback function and it's just going to dump all of the results into some data structure which is then going to be used by this write HTML function so that's the whole point there top key is just the argument so it's currently 50 which means we're gonna search for 50 closest matches inside of the index data set using the query vector and that's pretty much it on the high level before I show you how these post functions work let's step into the generator itself and see what's going on there so the generator is gonna iterate over all of the 60k docs as you can see here and it's gonna basically a load you can see here the structure of the dictionary we fetch the index data and then we just fetch the first image and for some reason they are inverting the image with 255 here I'm not sure why this is if somebody knows feel free to post to comment down below but they yeah for some reason they are they're inverting it and now what the next thing they're gonna do is because this is just a grayscale image they're gonna do stack times three here which is gonna form RGB image so they're basically gonna copy that channel three times and get an RGB image so now this is the document so this is the third core component of genome framework so I mentioned flow I mentioned executors so document is just like a wrapper object around all of different data types so here we're gonna form an image and so I'm gonna step over here and we're gonna set a tag ID to this this is gonna be zero because we are iterating remember over all of the 60k docs so that's it now we're gonna yield D which basically this is the generator syntax of Python I'm gonna not I'm not gonna dig into that a lot deeper but yeah so if I step over again it's gonna hit the same line again and you may wonder why is that so the thing is the way this is set up in the background is this generator is gonna yield hundred images and only then will the post function we just saw so the from the flow object is gonna execute well this thing is generating images let's return back to app I so it's gonna generate hundred images and the reason is as I already explained there is a hidden default argument that's hundred and after it's generated a hundred images is gonna call the index is gonna trigger and what happens once index is called is all of the executors which have these request methods defined all of these functions which are decorated with this at requests on index are gonna get cold and not only those functions but also the functions that don't have anything any on parameter here so we're gonna see what I mean by that so requests so I'm gonna put a breakpoint here requests we have this one is triggered on search and eval so we don't have to worry about it yet so this one is as you can see that doesn't have the on part so that means every single flow post function is gonna trigger this one so I have a breakpoint here and let's go further on this converter function is not cold so I'm gonna skip it and this last one is eval so it's not gonna be cold as well so the functions that they're going to get cold right now are the basically encode function and the index function so let's let's kind of step over here so I'm gonna hit that five that's gonna generate hundred images and the index function will be cold so f5 and as you can see here the encode function is the first one to be cold let's understand what's going on here so Doc's is just a array of hundred images remember that that's the images generated by the index generator so we're gonna fetch the content which is just is going to fetch all of the images we're gonna stack them and we're going to end up with hundred twenty four twenty four three the shape will be something like that I suspect so let's see the content part so the shape is hundred twenty eight twenty three because remember we have RGB amnest images we have hundreds of them because of the generator and that's it now we're gonna reshape them so gonna they're gonna extract only single channels so I don't know I'm not sure why we had to form RGB images when we were extracting single channel here but yeah whatever so basically we're reshaping these seven eight four because 24 squared equals that number so we're gonna just flatten it out and you can see here hundred seven hundred eighty four and now this is a funny part so the first thing they're gonna do is because these amnest images are in the 0 to 55 range so they're gonna normalize them to 0 1 range and then they're gonna do this multiplication with this other matrix and that's going to form the embedding so let me kind of explain what this actually does so first things first let's see the shape the shape will be of the so we have self other matrix let me find that one so it's as you can see here seven hundred eighty four sixty four so once we multiply once we multiply so we have remember we have hundred we have seven eighty four here so that's a that's this matrix and then we multiply that thing so let me put brackets here so we multiply it with seven eight four sixty four that means we're gonna convert this shape into hundred sixty four so we basically embedded all of those images into this 64 dimensional embedding vector so the logic here is not actually a neural network the thing they do is in the like constructor function they just form this random matrix that has exactly this shape then they apply the singular value decomposition I'm not gonna go to details here it's an animal technique and the thing you should probably know is that any matrix no matter the shape so it doesn't have to be a square matrix can be decomposition using SVD so once we have SVD that the nice property of these two matrices is that they are orthogonal that means once you multiply a vector by this matrix the vector is just gonna change the rotation but the length will stay the same so it's preserving the length of the vector so just rotation matrix and what I do is they multiply these two matrices to form this this other matrix so just a fancy way of creating an orthogonal matrix I the matrix that does rotations of this particular shape and that's gonna embed our vectors into this 64 dimensional vector that's it okay let's continue here so let's press F10 here so now the next thing that's gonna happen is we're gonna iterate we're gonna fill in these documents with embeddings and see here embeddings and the content content is just a flattened out amnesty image this function is gonna convert the actual blob the image into URIs which is unique resource identifier which is basically like a long string that's gonna be an equivalent representation of this image and then they're gonna pop the blob that's it we're gonna repeat this for all of the images and that's the encoding part of the index of this of this index function so remember we have hundred images here now we just what it we just did is we embedded all of these images and the next thing we're gonna do so I'm gonna just press F5 here we should hit one of the other functions so the the indexing function I guess so F5 and as you can see here the indexing function the only thing it does is it has this docs like private variable which is a document array mem a mem map and just another object from Gina which helps you kind of optimize memory wise and time wise the performance so I won't get into the details because it's just an optimization trick okay so we're gonna just dump all of those docs which now remember contain the embeddings as well as the original images convert it into your eyes I went ahead and executed the whole index function because it takes roughly 58 seconds on my machine so yeah so what happened is again we chunked the data the 60k images into hundred images per batch and we did the embeddings we did the extension of that specialized memory object so we just kind of added all of those documents with the embedding vectors to that specialized memory object the second thing that's gonna happen is we're gonna hold the post eval function and this again has this query generator so I'm gonna step into it and put a breakpoint here so put a breakpoint there return here so we're gonna step over and see what the query generator is doing and then we're gonna hit set the breakpoints to all of the executors which are kind of sign up to listen to this eval call and I've said the breakpoints to all of those eval functions already so let's just step over here so query generator you can see the number of queries 128 we have the target so that's a dictionary that contains all the data and with ground truth equals to true let's see what it means so query generator okay what this line will do is the following you will find because we have remember we have MNIST data set and we have like I think there are 10 classes inside of it so what is going to do is return this object which will contain the following thing a dictionary that will have a label and then all of the images from the index data set that have so all of the IDs that have label 0 and then the same thing for label 1 so IDs here etc so let me step over this code so f10 and see what GTS looks like so again and if I expand 0 somewhere in there you can find the information of the IDs of images or the document IDs so now a small drawback with Gina in my opinion is that it's kind of hard to find in these as you can see in variables while you're debugging the code to find the actual information you need because there is a lot of metadata and things you don't care about because of all of these complex document objects but it's also a good thing I guess at the end because it's wrapping up a lot of functionality for you so now we're gonna iterate through the like all of the docs we have 128 query vectors so we're gonna do so 128 times we're gonna form a random query vector so random ID from this because we have 10k query images remember we're gonna form a random ID here and we're gonna just fetch that like image that's again black and white first and then they form RGB image and then they just kind of wrap it up into the document and because the ground truth is set to true we're gonna return not only the query image but also the actual ground truth label this is again going to be called hundred times and only then will we have the post eval function being called as soon as gonna step over until we hit the first breakpoint and here is the funny part so basically what happens here is we the docs contains hundred query images and it's gonna do match function with this self dot underscore docs which if you remember contains the index images with all of the embedding vectors so what we're going to do here is for every single query image here we're gonna find the closest image in the index images using cosine metric just doing some normalization to zero from zero to one and finally we're gonna we're gonna fetch top K which means 50 closest images using the cosine like metric so in the background we're gonna have like a for loop simple like comparisons and at the end this talks container is gonna contain all of the matches inside of it gonna again step over and get to the next breakpoint finally we are here in the last executor so this here we're gonna calculate the precision and recall metrics so we've done the matching we now have for each very vector we have associated 50 like images from the index data set or from the training set and let's see some some information here so I can use VS codes nice functionality here and plot the number of documents we have so that's 28 and it's 28 because we have 128 remember and we're chunking into pieces of hundred and so 128 module 100 is gonna give us 28 query vectors in this iteration so ground shoots again contains the like basically the IDs of those images in the index data set that have the same label as a particular query vector so let me kind of break that down so let me step over here so first things first so this is a query vector and these are the matches from the like less function so basically these are the 50 images and we're gonna grab their IDs so we're gonna grab the IDs of those images from the index data set so let's see that so if I print actual here actual whoops actual okay so we can he see some numbers so those are the IDs of the images from the index data set now these are floats obviously there are some small inefficiencies and problems with this like example but it's working functionally it's working correctly so and finally desired contains the basically they said all of the IDs like that have the same label as this query vector so that means because we have 60k images and we have 10 labels I can expect to have 6,000 like IDs here so if I like kind of print length of desired like this whoops oh yeah I have to step over it so if I now try and print this it's 6,000 so yeah and now we can kind of do the intersection between the actual IDs we got and the desired so the ones that have the same label as the query vector and we can find the precision and recall metrics I'm gonna skip over that it's a simple simple logic and that's it now I'm going to basically remove this break a break point and jump to the final step of this whole program and that's writing all of this data into an HTML document and kind of rendering it inside of a browser okay here we are as I said we had this post eval a function being called and at the end we have had this callback function so let me kind of quickly walk you through what happened there so after we finished the calculation of like precision and recall metrics for every single query vector now this this function kind of populates this simple this is a simple like Python list it populates it with HTML syntax so we have as you see here here a table row table cell and we paste the like original query image and then we have we trade over all of the matches over the 50 matches we had all of those matches in the same row and then we close the cell we close the row and we repeat that for every single query vector that's 128 query vectors and that's how this thing is formed and after that let me kind of like jump into write HTML function I'm gonna put a breakpoint here and gonna press f5 so here we are it's gonna open up this demo HTML which is a template like HTML file and it's gonna replace some of the placeholder variables like as you can see here result we're gonna put this result HTML so that's the same just as Python string we just created that contains the content of the HTML file so yeah we're gonna replace all of these templates placeholders and finally we're gonna get let me jump to this web browser open it's gonna fail because I'm in WSL but that's just on my system it's gonna work for you much better and I'll just manually open it up so that's it when I press f5 this is the end of the program and I'm gonna open up the HTML okay here is the HTML let me open it up and here are the results and as you can see on the left hand side we've got query vectors on the right hand side we have the matches so the 50 matches for every single very vector and there is 128 of them and as you can see our results look pretty decent like if we focus on one particular image like I don't know like this shoe here we can see that most of these results are shoes they're not even though there are many other images in this data set like like bags or or whatever trousers I don't know there must be some like failure cases but I didn't manage to see anyone I don't see any failure case here in particular but yeah hopefully got some idea what Gina can do and let me just kind of mention that this second example with the QA the question-answer like chatbot is maybe a bit better because remember that Gina is a cloud native neural search so cloud and this example here what it does is it's very similar but like it opens up a chatbot so and it creates a Python server and so what happens is there is a communication happening between a Gina Python server and this JavaScript code in the browser and so you basically input like a query like a text query inside of the browser it gets embedded the Gina does its magic of finding the similar the most similar match so the most similar response the textual response and it communicates it back via the like this HTTPS I guess like a link to the JavaScript code and it just renders the the response it's a very similar example but it basically paints a clear picture the Gina is used for these cloud types of applications and because here we have explicitly like a Python server so hopefully found this video useful if you did share it out with a friend subscribe to the channel hit that bell icon also if you have any feedback on this video or what you would like me which ML tool you'd like me to cover next let me know down in the comments and also join the discord community until next time bye bye
[{"start": 0.0, "end": 4.68, "text": " What's cracking guys in this video I'm continuing with the coding video series"}, {"start": 4.68, "end": 9.5, "text": " and I'm not going to cover a research paper like code in this one I'm gonna"}, {"start": 9.5, "end": 14.76, "text": " cover open source ML tools and libraries and hopefully this will help you improve"}, {"start": 14.76, "end": 18.84, "text": " your own projects by including these awesome open source libraries and tools"}, {"start": 18.84, "end": 24.88, "text": " so in this one I'm gonna cover Gini AI and as you can see here they are a cloud"}, {"start": 24.88, "end": 28.8, "text": " native neural search framework for any kind of data so this enables it to"}, {"start": 28.8, "end": 33.0, "text": " import lightweight Google into your project whereas the neural part means"}, {"start": 33.0, "end": 37.760000000000005, "text": " you'll be using deep neural networks to perform the search and the any kind of"}, {"start": 37.760000000000005, "end": 41.24, "text": " data means basically you can take whatever modality you wish so like image"}, {"start": 41.24, "end": 46.28, "text": " or text or audio or video or mesh or whatnot and you can use those to query"}, {"start": 46.28, "end": 50.760000000000005, "text": " and find like data from any other modalities you can use images to query"}, {"start": 50.760000000000005, "end": 55.28, "text": " for certain text snippets you can use text to query for images or videos or"}, {"start": 55.28, "end": 59.160000000000004, "text": " audio for mesh or whatever like whatever combination in that combinatorial space"}, {"start": 59.160000000000004, "end": 64.96000000000001, "text": " you can imagine Gini AI can kind of help you do that as you can see the readme is"}, {"start": 64.96000000000001, "end": 70.0, "text": " pretty nicely structured they have lots of resources here to tutorials they have"}, {"start": 70.0, "end": 73.8, "text": " like lots of contributors people are using this already it's an established"}, {"start": 73.8, "end": 77.84, "text": " tool as you can see here used by 130 like organizations and people already and"}, {"start": 77.84, "end": 84.16, "text": " so the way I suggest you go ahead and explore this tool is by like"}, {"start": 84.16, "end": 88.32, "text": " understanding these quick demos and in this video I'm gonna walk you through"}, {"start": 88.32, "end": 91.52, "text": " step-by-step through the fashion image search aside from that you should"}, {"start": 91.52, "end": 95.02, "text": " probably go and understand the basic concepts of Gina so that's document"}, {"start": 95.02, "end": 99.16, "text": " executor and flow and they have really nice pages where they explain how you"}, {"start": 99.16, "end": 104.92, "text": " can use the document how you can use all of these like a native concepts to Gina"}, {"start": 104.92, "end": 110.36, "text": " so document is an abstraction they use to describe whatever data type you have"}, {"start": 110.36, "end": 114.0, "text": " so basically document can be an image document can be a text document can be"}, {"start": 114.0, "end": 117.68, "text": " video or whatever data type you have don't get confused by the terminology so"}, {"start": 117.68, "end": 121.92, "text": " document if you're like me a document is gonna associate you to text data"}, {"start": 121.92, "end": 127.56, "text": " textual data only and even the emoji here is associating us to to text but"}, {"start": 127.56, "end": 131.48, "text": " it's not it's a it's an abstraction object which wraps up any kind of data"}, {"start": 131.48, "end": 135.72, "text": " then they have executors which process the data and flows finally organize"}, {"start": 135.72, "end": 140.66, "text": " those executors into these kinds of pipelines as you can see here on the"}, {"start": 140.66, "end": 144.56, "text": " image a short note on the installation procedure because they are cloud native"}, {"start": 144.56, "end": 148.52, "text": " neural search engine that basically means because cloud is mostly Linux that"}, {"start": 148.52, "end": 153.2, "text": " means they support Gina for Linux pretty much and because I'm on Windows this"}, {"start": 153.2, "end": 157.2, "text": " thing does not work on Windows and I had to install a window subsystem for Linux"}, {"start": 157.2, "end": 162.24, "text": " I had some problems getting into work but after you have upgraded to WSL 2 and"}, {"start": 162.24, "end": 166.2, "text": " fix some other small problems like I had problems plotting because Matplotlib"}, {"start": 166.2, "end": 171.07999999999998, "text": " would not display the actual like chart so I had to do some tweaks so if you"}, {"start": 171.07999999999998, "end": 173.88, "text": " know you want to know all of those details if you are on Windows like me"}, {"start": 173.88, "end": 178.11999999999998, "text": " and you want to try this thing ago do join the discord community because I'm"}, {"start": 178.11999999999998, "end": 183.32, "text": " gonna gonna paste like the step-by-step how I kind of solved all of the problems"}, {"start": 183.32, "end": 187.72, "text": " I encountered on Windows and how I got it to work so having said that let's"}, {"start": 187.72, "end": 191.72, "text": " start with the fashion image search so they have a nice page here explaining"}, {"start": 191.72, "end": 198.04, "text": " how you can actually get it like running and the the end like result will be this"}, {"start": 198.04, "end": 202.07999999999998, "text": " you can see here basically we have a query image on the left and we can fetch"}, {"start": 202.07999999999998, "end": 206.88, "text": " like they are fetching 50 closest images from the data set where closeness is"}, {"start": 206.88, "end": 211.72, "text": " defined as you can see here like visual similarity in the case of images so yeah"}, {"start": 211.72, "end": 215.68, "text": " let's get this project started okay so the first thing you obviously need to do"}, {"start": 215.68, "end": 220.07999999999998, "text": " here is do git clone clone the repository on your own machine install"}, {"start": 220.08, "end": 226.44000000000003, "text": " the condo environment and just do pip install you Gina and that's gonna"}, {"start": 226.44000000000003, "end": 232.12, "text": " install Gina AI package to your environment and then you can get started"}, {"start": 232.12, "end": 237.66000000000003, "text": " so as you can see here I'm Windows so I'm in this like WSL terminal I'm using"}, {"start": 237.66000000000003, "end": 241.68, "text": " Ubuntu as my distribution of choice I'm just going to start Visual Studio code"}, {"start": 241.68, "end": 245.60000000000002, "text": " you can use whatever editor you prefer just like kind of navigate to the root"}, {"start": 245.6, "end": 251.32, "text": " directory of your project and again from your environment where you're using"}, {"start": 251.32, "end": 259.0, "text": " conda or something else just start your ID and then we can get started the first"}, {"start": 259.0, "end": 263.4, "text": " thing we will want to do is to clone the demo example to the root directory so"}, {"start": 263.4, "end": 271.56, "text": " I'm just gonna type Gina hello fork fashion into Gina fashion demo"}, {"start": 271.56, "end": 276.68, "text": " directory so basically this means hey I want to use Gina and I want to use these"}, {"start": 276.68, "end": 281.4, "text": " quick examples I want to fork it to my local machine I want to use fashion"}, {"start": 281.4, "end": 285.88, "text": " example in particular and I'm gonna fork it into Gina fashion demo so if I run"}, {"start": 285.88, "end": 291.8, "text": " that the as you can see here I just got this new folder downloaded and I'm gonna"}, {"start": 291.8, "end": 298.08, "text": " open up the the app.py so this is the main like main file and the thing we're"}, {"start": 298.08, "end": 301.32, "text": " going to do so we basically don't need any additional arguments so we can just"}, {"start": 301.32, "end": 308.32, "text": " start this thing and that's it so I'm gonna run f5 and start this project okay"}, {"start": 308.32, "end": 311.56, "text": " there are some basic parameters we are setting up here let me quickly just walk"}, {"start": 311.56, "end": 316.0, "text": " you through them so basically you can see here what is important is this part"}, {"start": 316.0, "end": 320.0, "text": " here so what they include they have this working directory the work directory is"}, {"start": 320.0, "end": 323.88, "text": " just going to create a random directory inside of your root directory and that's"}, {"start": 323.88, "end": 328.64, "text": " pretty much the only thing we care about in this one and then what they do is they"}, {"start": 328.64, "end": 335.71999999999997, "text": " as you can see here they have the data URL the the labels like for the training"}, {"start": 335.71999999999997, "end": 338.91999999999996, "text": " set so these two are for the training set so we are downloading fashion MNIST"}, {"start": 338.91999999999996, "end": 344.32, "text": " here and then they have the query data so basically the let's call these these"}, {"start": 344.32, "end": 349.8, "text": " are test sets again fashion MNIST and we have the data and we have the the labels"}, {"start": 349.8, "end": 354.08, "text": " here so number queries is just the number of queries that we're gonna"}, {"start": 354.08, "end": 359.56, "text": " display on the on the web page we'll see soon see and finally top k means how"}, {"start": 359.56, "end": 363.8, "text": " many how many similar items are we trying to find for each of these queries"}, {"start": 363.8, "end": 368.0, "text": " so those are all of the parameters you should care about and now let's return"}, {"start": 368.0, "end": 373.03999999999996, "text": " to the app.py so let's kind of step into this function and see what's going on so"}, {"start": 373.03999999999996, "end": 378.28, "text": " first things first they are going to create this random directory so that's"}, {"start": 378.28, "end": 384.0, "text": " it and then they're gonna form this this basically dictionary where they have as"}, {"start": 384.0, "end": 388.48, "text": " you can see here so the URL for the data and the local file name name where this"}, {"start": 388.48, "end": 393.6, "text": " data is gonna be saved so we're gonna save the the the training data directly"}, {"start": 393.6, "end": 398.48, "text": " inside of this random directory and we're gonna name this index labels file"}, {"start": 398.48, "end": 403.2, "text": " so nothing special here we're gonna do that for both the training set and the"}, {"start": 403.2, "end": 408.44, "text": " test set and that's it okay let's continue so next step is downloading the"}, {"start": 408.44, "end": 414.4, "text": " data so let's step into this download data function so basically nothing smart"}, {"start": 414.4, "end": 420.2, "text": " here this is everything is handled by this URL lib package so what happens is"}, {"start": 420.2, "end": 424.28, "text": " we're gonna skip this because it's none so we're gonna install this opener"}, {"start": 424.28, "end": 426.92, "text": " whatever so this is just like boilerplate pretty much you don't need"}, {"start": 426.92, "end": 430.84, "text": " to understand all of these details with progress bar just a context manager to"}, {"start": 430.84, "end": 434.36, "text": " show the progress while the thing is being downloaded and now we're gonna"}, {"start": 434.36, "end": 438.15999999999997, "text": " iterate through the dictionary and download the data and the labels for the"}, {"start": 438.16, "end": 442.6, "text": " training set and for the test set that's it so if it if it doesn't exist and it"}, {"start": 442.6, "end": 445.92, "text": " doesn't exist because we just created this random folder we're gonna retrieve"}, {"start": 445.92, "end": 450.88000000000005, "text": " it here we're gonna download it so until this point we just downloaded the data"}, {"start": 450.88000000000005, "end": 454.88, "text": " into those files and now we're gonna actually load the data from the files"}, {"start": 454.88, "end": 459.92, "text": " let's see load labels let's step into that one and it's very simple what they"}, {"start": 459.92, "end": 464.08000000000004, "text": " do is just they just open up that like local file and let me show you where it"}, {"start": 464.08000000000004, "end": 466.68, "text": " is so basically this is the random directory I was mentioning so it's"}, {"start": 466.68, "end": 470.52, "text": " directly inside of Gina root you open it up and you can see index label so that's"}, {"start": 470.52, "end": 474.16, "text": " the first file so now here we're gonna open it up as you can see here and we're"}, {"start": 474.16, "end": 481.16, "text": " gonna load it from using a numpy and just reading the file here so nothing"}, {"start": 481.16, "end": 484.88, "text": " you need to care about so this is some specific knowledge because they know the"}, {"start": 484.88, "end": 488.68, "text": " structure of this file they know that they have to offset reading eight bytes"}, {"start": 488.68, "end": 492.72, "text": " from the very start and that's it then they do the reshape part and this is"}, {"start": 492.72, "end": 495.96000000000004, "text": " always useful to have shapes in your mind constantly when you're debugging"}, {"start": 495.96, "end": 501.59999999999997, "text": " machine learning code so this should be a 60 K comma one should be the shape of"}, {"start": 501.59999999999997, "end": 507.03999999999996, "text": " this of this object because those are labels and we have 60 K examples inside"}, {"start": 507.03999999999996, "end": 511.02, "text": " of the index data set so I'm gonna step over here and we can check the data to"}, {"start": 511.02, "end": 515.68, "text": " see whether we are correct here so V I'm gonna find the data array and you can"}, {"start": 515.68, "end": 523.4, "text": " see here shape is 60 K comma one so that's it now we're gonna do the same"}, {"start": 523.4, "end": 529.04, "text": " thing for for the next label so this is gonna download the labels for the query"}, {"start": 529.04, "end": 533.4399999999999, "text": " data set same procedure so I won't be walking you through the same function"}, {"start": 533.4399999999999, "end": 537.92, "text": " again and finally we have we're downloading the actual images through"}, {"start": 537.92, "end": 541.6, "text": " that's done you can see in the directory we have this index originals that means"}, {"start": 541.6, "end": 546.88, "text": " these are actually the images okay and now we're gonna step into this load"}, {"start": 546.88, "end": 551.0799999999999, "text": " MNIST I'm gonna step into it and you can see again we just open up the file we"}, {"start": 551.08, "end": 557.24, "text": " read it we like have this non pipe from buffer and we reshape it into 60 K 28 by"}, {"start": 557.24, "end": 561.96, "text": " 28 because we have MNIST images here so that's it it's fairly simple I'm gonna"}, {"start": 561.96, "end": 570.44, "text": " just step over this and we're gonna see that after I do this so we data and we"}, {"start": 570.44, "end": 575.8000000000001, "text": " have 60 K 28 28 those are obviously MNIST images and finally last step we're"}, {"start": 575.8000000000001, "end": 578.8000000000001, "text": " gonna do the same thing so we're doing the same thing just for the query data"}, {"start": 578.8, "end": 583.1999999999999, "text": " set this time and I'm gonna step over this time so we're gonna load the data"}, {"start": 583.1999999999999, "end": 588.68, "text": " and that's it we downloaded the data we loaded the labels we loaded the like"}, {"start": 588.68, "end": 592.4399999999999, "text": " images and that's it so that data is currently stored into this dictionary so"}, {"start": 592.4399999999999, "end": 596.52, "text": " we have targets you can pick whatever type of data you want so for example we"}, {"start": 596.52, "end": 600.4799999999999, "text": " can we can pick index labels so that means the index data set the labels and"}, {"start": 600.4799999999999, "end": 605.88, "text": " you'll find the data array here so it's 60 K 1 you can do the same thing for"}, {"start": 605.88, "end": 611.68, "text": " example index which is images and you can see the data 60 K 28 28 so"}, {"start": 611.68, "end": 614.96, "text": " everything is stored into this neatly stored into this dictionary object and"}, {"start": 614.96, "end": 619.32, "text": " that's it now this is some nothing that important just some optimization stuff"}, {"start": 619.32, "end": 623.0, "text": " and finally this is the flow object I mentioned in the initial part of the"}, {"start": 623.0, "end": 628.64, "text": " video which is one of the like building blocks of these GNA like programs and"}, {"start": 628.64, "end": 634.16, "text": " what flow does is just organizes these executors into this nice pipeline which"}, {"start": 634.16, "end": 638.24, "text": " you can then run and execute a process the data in whichever whatever manner"}, {"start": 638.24, "end": 644.92, "text": " you wish so as you can see here we're not form flow and we're gonna add these"}, {"start": 644.92, "end": 649.0, "text": " so these are called executors so we have executor that's called my encoder we're"}, {"start": 649.0, "end": 653.0799999999999, "text": " gonna see how that thing what the thing does in a couple of minutes we have we'll"}, {"start": 653.0799999999999, "end": 658.36, "text": " have two of those in parallel then we'll have the indexer and the indexer is"}, {"start": 658.36, "end": 662.8399999999999, "text": " gonna index this thing into this rug directory which is the random UUID"}, {"start": 662.84, "end": 666.0400000000001, "text": " directory as you can see here the string on the screen and finally we have this"}, {"start": 666.0400000000001, "end": 672.0400000000001, "text": " my evaluator executor which is going to do some the actual searching and matching"}, {"start": 672.0400000000001, "end": 677.72, "text": " of the image embedding vector of the query embedding vector to the all of"}, {"start": 677.72, "end": 682.24, "text": " the other index embedding vectors so that's that's it and now because this is"}, {"start": 682.24, "end": 686.96, "text": " lazy execution this object still the constructors of these executors are"}, {"start": 686.96, "end": 689.9200000000001, "text": " still not called that's gonna happen after we open it up with a context"}, {"start": 689.92, "end": 694.76, "text": " manager here so only here will they actually execute and form those those"}, {"start": 694.76, "end": 700.0, "text": " like executors so that's the lazy execution design pattern so the thing"}, {"start": 700.0, "end": 704.24, "text": " I'm gonna do now is actually visualize this flow so I'm gonna stop this program"}, {"start": 704.24, "end": 708.3199999999999, "text": " I'm gonna add the plotting function here so we can plot you can actually plot"}, {"start": 708.3199999999999, "end": 712.7199999999999, "text": " these like flows and we're gonna just just save it into like I don't know like"}, {"start": 712.7199999999999, "end": 717.8, "text": " time flow SVG and that's it I'm gonna save this I'm gonna rerun it and I'm"}, {"start": 717.8, "end": 722.04, "text": " gonna show you what happens so we are here if I do a step over let's just step"}, {"start": 722.04, "end": 726.04, "text": " over it's gonna form this SVG and I'm gonna open it up and show you how this"}, {"start": 726.04, "end": 730.92, "text": " thing looks like here it is it's you can see the temporary flow SVG file in the"}, {"start": 730.92, "end": 734.52, "text": " root directory and for some reason it's gonna open it up in Microsoft Edge even"}, {"start": 734.52, "end": 737.88, "text": " though my default browser is Google Chrome so let's open it up here and you"}, {"start": 737.88, "end": 745.2199999999999, "text": " can see the actual like diagram so we have the input here we split the input"}, {"start": 745.22, "end": 749.4, "text": " into two my encoder executors then we have the my indexer and then we have the"}, {"start": 749.4, "end": 754.88, "text": " my evaluator if we were to put some names as the arguments to those here"}, {"start": 754.88, "end": 760.08, "text": " basically name they have a name parameter blah blah blah we have nicer"}, {"start": 760.08, "end": 763.44, "text": " diagrams but you get the point okay having done that let me now show you"}, {"start": 763.44, "end": 766.88, "text": " what I meant by delays execution so we're here now I still haven't executed"}, {"start": 766.88, "end": 772.36, "text": " this line so if I go to let me find the files let me find the the actual my"}, {"start": 772.36, "end": 776.08, "text": " executors folder so I'm gonna put some breakpoint inside of the constructors"}, {"start": 776.08, "end": 780.96, "text": " here so we need functions my indexer I'm gonna put a breaking point here my"}, {"start": 780.96, "end": 785.76, "text": " encoder and finally I'm gonna put a breaking point into my evaluator and so"}, {"start": 785.76, "end": 789.76, "text": " now I'm gonna put a breakpoint here and I'm gonna step over this line here and"}, {"start": 789.76, "end": 793.72, "text": " we're gonna see it's gonna hit the constructors of those executors so f10"}, {"start": 793.72, "end": 801.3000000000001, "text": " and you can see here my encoder is hit here so I'm gonna do f5 again we we had"}, {"start": 801.3, "end": 805.68, "text": " encoder again because we have is remember two encoder executors in"}, {"start": 805.68, "end": 812.52, "text": " parallel so I'm gonna step over f10 so f5 indexer and f5 evaluator and that's"}, {"start": 812.52, "end": 817.68, "text": " it f5 will bring us to the next line f index okay so you saw that by doing this"}, {"start": 817.68, "end": 822.68, "text": " we only then executed the constructors of all of the executors you've inputted"}, {"start": 822.68, "end": 828.0799999999999, "text": " here and now now we're gonna hit the index function so first let me break"}, {"start": 828.08, "end": 831.44, "text": " this down on a high level for you and then we're gonna step over the code so"}, {"start": 831.44, "end": 835.1600000000001, "text": " we have as you can see here the main function there is not much lines of"}, {"start": 835.1600000000001, "end": 839.2, "text": " code here left it's fairly simple we have we're calling flow index which is"}, {"start": 839.2, "end": 844.36, "text": " the same thing as if if we were to call this post function and instead of using"}, {"start": 844.36, "end": 849.1600000000001, "text": " slash eval here we'd be using slash index so these are the same things so"}, {"start": 849.1600000000001, "end": 854.76, "text": " the indexing function is gonna as you can imagine kind of index the data and"}, {"start": 854.76, "end": 858.76, "text": " gonna create the embedding vectors and then the eval function is gonna do the"}, {"start": 858.76, "end": 862.3199999999999, "text": " actual matching so once we call the eval function we're gonna actually find the"}, {"start": 862.3199999999999, "end": 867.4399999999999, "text": " 50 closest matches to that query image okay so that's like the rough pipeline"}, {"start": 867.4399999999999, "end": 871.48, "text": " and then finally after we have the data we're gonna write it into HTML and we're"}, {"start": 871.48, "end": 876.0, "text": " gonna be able to open up this HTML page in the browser and see the final results"}, {"start": 876.0, "end": 880.6, "text": " let's go a bit deeper and then I'm gonna step into these this code so next thing"}, {"start": 880.6, "end": 883.84, "text": " you see we have these generators we have index generator here we have query"}, {"start": 883.84, "end": 887.44, "text": " generator here and we have some additional parameters not that important"}, {"start": 887.44, "end": 891.5600000000001, "text": " show progress is just gonna show us in the bar as the testing is being indexed"}, {"start": 891.5600000000001, "end": 896.0, "text": " and evaluated we're gonna see the progress on the in the console print"}, {"start": 896.0, "end": 900.0, "text": " results so once this thing is done the print result is a callback function and"}, {"start": 900.0, "end": 904.44, "text": " it's just going to dump all of the results into some data structure which"}, {"start": 904.44, "end": 908.36, "text": " is then going to be used by this write HTML function so that's the whole point"}, {"start": 908.36, "end": 912.64, "text": " there top key is just the argument so it's currently 50 which means we're"}, {"start": 912.64, "end": 917.4399999999999, "text": " gonna search for 50 closest matches inside of the index data set using the"}, {"start": 917.4399999999999, "end": 922.1999999999999, "text": " query vector and that's pretty much it on the high level before I show you how"}, {"start": 922.1999999999999, "end": 927.08, "text": " these post functions work let's step into the generator itself and see what's"}, {"start": 927.08, "end": 932.1999999999999, "text": " going on there so the generator is gonna iterate over all of the 60k docs as"}, {"start": 932.1999999999999, "end": 936.1999999999999, "text": " you can see here and it's gonna basically a load you can see here the"}, {"start": 936.1999999999999, "end": 940.68, "text": " structure of the dictionary we fetch the index data and then we just fetch the"}, {"start": 940.68, "end": 945.3599999999999, "text": " first image and for some reason they are inverting the image with 255 here I'm"}, {"start": 945.3599999999999, "end": 949.12, "text": " not sure why this is if somebody knows feel free to post to comment down below"}, {"start": 949.12, "end": 953.68, "text": " but they yeah for some reason they are they're inverting it and now what the"}, {"start": 953.68, "end": 956.76, "text": " next thing they're gonna do is because this is just a grayscale image they're"}, {"start": 956.76, "end": 961.2399999999999, "text": " gonna do stack times three here which is gonna form RGB image so they're"}, {"start": 961.2399999999999, "end": 967.4399999999999, "text": " basically gonna copy that channel three times and get an RGB image so now this"}, {"start": 967.44, "end": 972.6, "text": " is the document so this is the third core component of genome framework so I"}, {"start": 972.6, "end": 977.96, "text": " mentioned flow I mentioned executors so document is just like a wrapper object"}, {"start": 977.96, "end": 983.2800000000001, "text": " around all of different data types so here we're gonna form an image and so"}, {"start": 983.2800000000001, "end": 988.5600000000001, "text": " I'm gonna step over here and we're gonna set a tag ID to this this is gonna be"}, {"start": 988.5600000000001, "end": 992.32, "text": " zero because we are iterating remember over all of the 60k docs so that's it"}, {"start": 992.32, "end": 996.36, "text": " now we're gonna yield D which basically this is the generator syntax of Python"}, {"start": 996.36, "end": 1000.64, "text": " I'm gonna not I'm not gonna dig into that a lot deeper but yeah so if I step"}, {"start": 1000.64, "end": 1004.78, "text": " over again it's gonna hit the same line again and you may wonder why is that so"}, {"start": 1004.78, "end": 1007.84, "text": " the thing is the way this is set up in the background is this generator is"}, {"start": 1007.84, "end": 1013.22, "text": " gonna yield hundred images and only then will the post function we just saw so"}, {"start": 1013.22, "end": 1017.0, "text": " the from the flow object is gonna execute well this thing is generating"}, {"start": 1017.0, "end": 1021.32, "text": " images let's return back to app I so it's gonna generate hundred images and"}, {"start": 1021.32, "end": 1025.08, "text": " the reason is as I already explained there is a hidden default argument"}, {"start": 1025.08, "end": 1029.0, "text": " that's hundred and after it's generated a hundred images is gonna call the index"}, {"start": 1029.0, "end": 1033.1599999999999, "text": " is gonna trigger and what happens once index is called is all of the executors"}, {"start": 1033.1599999999999, "end": 1036.9199999999998, "text": " which have these request methods defined all of these functions which are"}, {"start": 1036.9199999999998, "end": 1042.8999999999999, "text": " decorated with this at requests on index are gonna get cold and not only those"}, {"start": 1042.8999999999999, "end": 1046.8, "text": " functions but also the functions that don't have anything any on parameter"}, {"start": 1046.8, "end": 1049.6399999999999, "text": " here so we're gonna see what I mean by that so requests so I'm gonna put a"}, {"start": 1049.6399999999999, "end": 1053.8799999999999, "text": " breakpoint here requests we have this one is triggered on search and eval so"}, {"start": 1053.88, "end": 1057.88, "text": " we don't have to worry about it yet so this one is as you can see that doesn't"}, {"start": 1057.88, "end": 1062.3600000000001, "text": " have the on part so that means every single flow post function is gonna"}, {"start": 1062.3600000000001, "end": 1066.1200000000001, "text": " trigger this one so I have a breakpoint here and let's go further on this"}, {"start": 1066.1200000000001, "end": 1070.24, "text": " converter function is not cold so I'm gonna skip it and this last one is eval"}, {"start": 1070.24, "end": 1074.24, "text": " so it's not gonna be cold as well so the functions that they're going to get"}, {"start": 1074.24, "end": 1080.7600000000002, "text": " cold right now are the basically encode function and the index function so let's"}, {"start": 1080.76, "end": 1084.96, "text": " let's kind of step over here so I'm gonna hit that five that's gonna"}, {"start": 1084.96, "end": 1090.12, "text": " generate hundred images and the index function will be cold so f5 and as you"}, {"start": 1090.12, "end": 1093.92, "text": " can see here the encode function is the first one to be cold let's understand"}, {"start": 1093.92, "end": 1098.56, "text": " what's going on here so Doc's is just a array of hundred images remember that"}, {"start": 1098.56, "end": 1102.4, "text": " that's the images generated by the index generator so we're gonna fetch the"}, {"start": 1102.4, "end": 1106.48, "text": " content which is just is going to fetch all of the images we're gonna stack them"}, {"start": 1106.48, "end": 1111.2, "text": " and we're going to end up with hundred twenty four twenty four three the shape"}, {"start": 1111.2, "end": 1115.76, "text": " will be something like that I suspect so let's see the content part so the shape"}, {"start": 1115.76, "end": 1119.1200000000001, "text": " is hundred twenty eight twenty three because remember we have RGB amnest"}, {"start": 1119.1200000000001, "end": 1122.92, "text": " images we have hundreds of them because of the generator and that's it now we're"}, {"start": 1122.92, "end": 1126.24, "text": " gonna reshape them so gonna they're gonna extract only single channels so I"}, {"start": 1126.24, "end": 1131.56, "text": " don't know I'm not sure why we had to form RGB images when we were extracting"}, {"start": 1131.56, "end": 1136.04, "text": " single channel here but yeah whatever so basically we're reshaping these seven"}, {"start": 1136.04, "end": 1140.1599999999999, "text": " eight four because 24 squared equals that number so we're gonna just flatten"}, {"start": 1140.1599999999999, "end": 1144.1599999999999, "text": " it out and you can see here hundred seven hundred eighty four and now this"}, {"start": 1144.1599999999999, "end": 1148.68, "text": " is a funny part so the first thing they're gonna do is because these amnest"}, {"start": 1148.68, "end": 1154.48, "text": " images are in the 0 to 55 range so they're gonna normalize them to 0 1 range"}, {"start": 1154.48, "end": 1158.32, "text": " and then they're gonna do this multiplication with this other matrix"}, {"start": 1158.32, "end": 1161.96, "text": " and that's going to form the embedding so let me kind of explain what this"}, {"start": 1161.96, "end": 1165.36, "text": " actually does so first things first let's see the shape the shape will be of"}, {"start": 1165.36, "end": 1169.8799999999999, "text": " the so we have self other matrix let me find that one so it's as you can see"}, {"start": 1169.8799999999999, "end": 1174.12, "text": " here seven hundred eighty four sixty four so once we multiply once we"}, {"start": 1174.12, "end": 1178.28, "text": " multiply so we have remember we have hundred we have seven eighty four here"}, {"start": 1178.28, "end": 1182.24, "text": " so that's a that's this matrix and then we multiply that thing so let me put"}, {"start": 1182.24, "end": 1188.32, "text": " brackets here so we multiply it with seven eight four sixty four that means"}, {"start": 1188.32, "end": 1192.8799999999999, "text": " we're gonna convert this shape into hundred sixty four so we basically"}, {"start": 1192.88, "end": 1199.16, "text": " embedded all of those images into this 64 dimensional embedding vector so the"}, {"start": 1199.16, "end": 1202.88, "text": " logic here is not actually a neural network the thing they do is in the"}, {"start": 1202.88, "end": 1208.0, "text": " like constructor function they just form this random matrix that has exactly"}, {"start": 1208.0, "end": 1213.0800000000002, "text": " this shape then they apply the singular value decomposition I'm not gonna go to"}, {"start": 1213.0800000000002, "end": 1216.24, "text": " details here it's an animal technique and the thing you should probably know"}, {"start": 1216.24, "end": 1220.3600000000001, "text": " is that any matrix no matter the shape so it doesn't have to be a square matrix"}, {"start": 1220.36, "end": 1225.3999999999999, "text": " can be decomposition using SVD so once we have SVD that the nice property of"}, {"start": 1225.3999999999999, "end": 1229.6799999999998, "text": " these two matrices is that they are orthogonal that means once you multiply"}, {"start": 1229.6799999999998, "end": 1234.28, "text": " a vector by this matrix the vector is just gonna change the rotation but the"}, {"start": 1234.28, "end": 1237.52, "text": " length will stay the same so it's preserving the length of the vector so"}, {"start": 1237.52, "end": 1242.1999999999998, "text": " just rotation matrix and what I do is they multiply these two matrices to form"}, {"start": 1242.1999999999998, "end": 1246.4799999999998, "text": " this this other matrix so just a fancy way of creating an orthogonal matrix I"}, {"start": 1246.48, "end": 1251.6, "text": " the matrix that does rotations of this particular shape and that's gonna embed"}, {"start": 1251.6, "end": 1255.8, "text": " our vectors into this 64 dimensional vector that's it okay let's continue"}, {"start": 1255.8, "end": 1261.48, "text": " here so let's press F10 here so now the next thing that's gonna happen is we're"}, {"start": 1261.48, "end": 1265.24, "text": " gonna iterate we're gonna fill in these documents with embeddings and see here"}, {"start": 1265.24, "end": 1269.06, "text": " embeddings and the content content is just a flattened out amnesty image this"}, {"start": 1269.06, "end": 1274.56, "text": " function is gonna convert the actual blob the image into URIs which is unique"}, {"start": 1274.56, "end": 1279.28, "text": " resource identifier which is basically like a long string that's gonna be an"}, {"start": 1279.28, "end": 1283.8, "text": " equivalent representation of this image and then they're gonna pop the blob"}, {"start": 1283.8, "end": 1287.72, "text": " that's it we're gonna repeat this for all of the images and that's the"}, {"start": 1287.72, "end": 1293.04, "text": " encoding part of the index of this of this index function so remember we have"}, {"start": 1293.04, "end": 1298.32, "text": " hundred images here now we just what it we just did is we embedded all of these"}, {"start": 1298.32, "end": 1303.1599999999999, "text": " images and the next thing we're gonna do so I'm gonna just press F5 here we"}, {"start": 1303.16, "end": 1309.28, "text": " should hit one of the other functions so the the indexing function I guess so F5"}, {"start": 1309.28, "end": 1313.1200000000001, "text": " and as you can see here the indexing function the only thing it does is it"}, {"start": 1313.1200000000001, "end": 1318.92, "text": " has this docs like private variable which is a document array mem a mem map"}, {"start": 1318.92, "end": 1323.92, "text": " and just another object from Gina which helps you kind of optimize memory wise"}, {"start": 1323.92, "end": 1327.0400000000002, "text": " and time wise the performance so I won't get into the details because it's just"}, {"start": 1327.0400000000002, "end": 1331.68, "text": " an optimization trick okay so we're gonna just dump all of those docs which"}, {"start": 1331.68, "end": 1336.1200000000001, "text": " now remember contain the embeddings as well as the original images convert it"}, {"start": 1336.1200000000001, "end": 1339.5600000000002, "text": " into your eyes I went ahead and executed the whole index function because it"}, {"start": 1339.5600000000002, "end": 1344.04, "text": " takes roughly 58 seconds on my machine so yeah so what happened is again we"}, {"start": 1344.04, "end": 1350.0600000000002, "text": " chunked the data the 60k images into hundred images per batch and we did the"}, {"start": 1350.0600000000002, "end": 1355.0800000000002, "text": " embeddings we did the extension of that specialized memory object so we just"}, {"start": 1355.0800000000002, "end": 1358.3200000000002, "text": " kind of added all of those documents with the embedding vectors to that"}, {"start": 1358.32, "end": 1362.4399999999998, "text": " specialized memory object the second thing that's gonna happen is we're gonna"}, {"start": 1362.4399999999998, "end": 1367.28, "text": " hold the post eval function and this again has this query generator so I'm"}, {"start": 1367.28, "end": 1372.2, "text": " gonna step into it and put a breakpoint here so put a breakpoint there return"}, {"start": 1372.2, "end": 1375.12, "text": " here so we're gonna step over and see what the query generator is doing and"}, {"start": 1375.12, "end": 1379.32, "text": " then we're gonna hit set the breakpoints to all of the executors which are kind"}, {"start": 1379.32, "end": 1384.28, "text": " of sign up to listen to this eval call and I've said the breakpoints to all of"}, {"start": 1384.28, "end": 1388.12, "text": " those eval functions already so let's just step over here so query generator"}, {"start": 1388.12, "end": 1392.12, "text": " you can see the number of queries 128 we have the target so that's a dictionary"}, {"start": 1392.12, "end": 1396.1999999999998, "text": " that contains all the data and with ground truth equals to true let's see"}, {"start": 1396.1999999999998, "end": 1400.9199999999998, "text": " what it means so query generator okay what this line will do is the following"}, {"start": 1400.9199999999998, "end": 1405.28, "text": " you will find because we have remember we have MNIST data set and we have like"}, {"start": 1405.28, "end": 1409.3999999999999, "text": " I think there are 10 classes inside of it so what is going to do is return this"}, {"start": 1409.3999999999999, "end": 1412.76, "text": " object which will contain the following thing a dictionary that will have a"}, {"start": 1412.76, "end": 1417.56, "text": " label and then all of the images from the index data set that have so all of"}, {"start": 1417.56, "end": 1423.36, "text": " the IDs that have label 0 and then the same thing for label 1 so IDs here etc"}, {"start": 1423.36, "end": 1428.9199999999998, "text": " so let me step over this code so f10 and see what GTS looks like so again and if"}, {"start": 1428.9199999999998, "end": 1432.9199999999998, "text": " I expand 0 somewhere in there you can find the information of the IDs of"}, {"start": 1432.9199999999998, "end": 1436.6399999999999, "text": " images or the document IDs so now a small drawback with Gina in my opinion"}, {"start": 1436.6399999999999, "end": 1440.48, "text": " is that it's kind of hard to find in these as you can see in variables while"}, {"start": 1440.48, "end": 1443.6, "text": " you're debugging the code to find the actual information you need because"}, {"start": 1443.6, "end": 1447.32, "text": " there is a lot of metadata and things you don't care about because of all of"}, {"start": 1447.32, "end": 1451.1599999999999, "text": " these complex document objects but it's also a good thing I guess at the end"}, {"start": 1451.1599999999999, "end": 1455.3999999999999, "text": " because it's wrapping up a lot of functionality for you so now we're gonna"}, {"start": 1455.3999999999999, "end": 1462.56, "text": " iterate through the like all of the docs we have 128 query vectors so we're gonna"}, {"start": 1462.56, "end": 1467.04, "text": " do so 128 times we're gonna form a random query vector so random ID from"}, {"start": 1467.04, "end": 1471.6399999999999, "text": " this because we have 10k query images remember we're gonna form a random ID"}, {"start": 1471.6399999999999, "end": 1476.6, "text": " here and we're gonna just fetch that like image that's again black and white"}, {"start": 1476.6, "end": 1479.9599999999998, "text": " first and then they form RGB image and then they just kind of wrap it up into"}, {"start": 1479.9599999999998, "end": 1483.8799999999999, "text": " the document and because the ground truth is set to true we're gonna return"}, {"start": 1483.8799999999999, "end": 1489.7199999999998, "text": " not only the query image but also the actual ground truth label this is again"}, {"start": 1489.7199999999998, "end": 1493.6399999999999, "text": " going to be called hundred times and only then will we have the post eval"}, {"start": 1493.6399999999999, "end": 1496.6399999999999, "text": " function being called as soon as gonna step over until we hit the first"}, {"start": 1496.6399999999999, "end": 1500.32, "text": " breakpoint and here is the funny part so basically what happens here is we the"}, {"start": 1500.32, "end": 1504.8799999999999, "text": " docs contains hundred query images and it's gonna do match function with this"}, {"start": 1504.88, "end": 1510.3600000000001, "text": " self dot underscore docs which if you remember contains the index images with"}, {"start": 1510.3600000000001, "end": 1514.92, "text": " all of the embedding vectors so what we're going to do here is for every"}, {"start": 1514.92, "end": 1520.2, "text": " single query image here we're gonna find the closest image in the index images"}, {"start": 1520.2, "end": 1525.3200000000002, "text": " using cosine metric just doing some normalization to zero from zero to one"}, {"start": 1525.3200000000002, "end": 1528.96, "text": " and finally we're gonna we're gonna fetch top K which means 50 closest"}, {"start": 1528.96, "end": 1534.0800000000002, "text": " images using the cosine like metric so in the background we're gonna have like"}, {"start": 1534.08, "end": 1538.84, "text": " a for loop simple like comparisons and at the end this talks container is gonna"}, {"start": 1538.84, "end": 1543.08, "text": " contain all of the matches inside of it gonna again step over and get to the"}, {"start": 1543.08, "end": 1547.9199999999998, "text": " next breakpoint finally we are here in the last executor so this here we're"}, {"start": 1547.9199999999998, "end": 1551.0, "text": " gonna calculate the precision and recall metrics so we've done the matching we"}, {"start": 1551.0, "end": 1556.84, "text": " now have for each very vector we have associated 50 like images from the index"}, {"start": 1556.84, "end": 1560.56, "text": " data set or from the training set and let's see some some information here so"}, {"start": 1560.56, "end": 1564.48, "text": " I can use VS codes nice functionality here and plot the number of documents we"}, {"start": 1564.48, "end": 1569.04, "text": " have so that's 28 and it's 28 because we have 128 remember and we're chunking"}, {"start": 1569.04, "end": 1574.96, "text": " into pieces of hundred and so 128 module 100 is gonna give us 28 query vectors in"}, {"start": 1574.96, "end": 1581.08, "text": " this iteration so ground shoots again contains the like basically the IDs of"}, {"start": 1581.08, "end": 1584.96, "text": " those images in the index data set that have the same label as a particular"}, {"start": 1584.96, "end": 1589.8799999999999, "text": " query vector so let me kind of break that down so let me step over here so"}, {"start": 1589.88, "end": 1593.5600000000002, "text": " first things first so this is a query vector and these are the matches from"}, {"start": 1593.5600000000002, "end": 1597.64, "text": " the like less function so basically these are the 50 images and we're gonna"}, {"start": 1597.64, "end": 1602.0, "text": " grab their IDs so we're gonna grab the IDs of those images from the index data"}, {"start": 1602.0, "end": 1609.3600000000001, "text": " set so let's see that so if I print actual here actual whoops actual okay so"}, {"start": 1609.3600000000001, "end": 1613.3600000000001, "text": " we can he see some numbers so those are the IDs of the images from the index"}, {"start": 1613.3600000000001, "end": 1617.0400000000002, "text": " data set now these are floats obviously there are some small inefficiencies and"}, {"start": 1617.04, "end": 1620.6, "text": " problems with this like example but it's working functionally it's working"}, {"start": 1620.6, "end": 1626.6399999999999, "text": " correctly so and finally desired contains the basically they said all of"}, {"start": 1626.6399999999999, "end": 1631.36, "text": " the IDs like that have the same label as this query vector so that means because"}, {"start": 1631.36, "end": 1638.04, "text": " we have 60k images and we have 10 labels I can expect to have 6,000 like IDs here"}, {"start": 1638.04, "end": 1644.68, "text": " so if I like kind of print length of desired like this whoops oh yeah I have"}, {"start": 1644.68, "end": 1651.8, "text": " to step over it so if I now try and print this it's 6,000 so yeah and now we"}, {"start": 1651.8, "end": 1656.4, "text": " can kind of do the intersection between the actual IDs we got and the desired so"}, {"start": 1656.4, "end": 1660.1200000000001, "text": " the ones that have the same label as the query vector and we can find the"}, {"start": 1660.1200000000001, "end": 1664.2, "text": " precision and recall metrics I'm gonna skip over that it's a simple simple"}, {"start": 1664.2, "end": 1667.72, "text": " logic and that's it now I'm going to basically remove this break a break point"}, {"start": 1667.72, "end": 1673.4, "text": " and jump to the final step of this whole program and that's writing all of this"}, {"start": 1673.4, "end": 1678.16, "text": " data into an HTML document and kind of rendering it inside of a browser okay"}, {"start": 1678.16, "end": 1684.5600000000002, "text": " here we are as I said we had this post eval a function being called and at the"}, {"start": 1684.5600000000002, "end": 1687.68, "text": " end we have had this callback function so let me kind of quickly walk you"}, {"start": 1687.68, "end": 1692.48, "text": " through what happened there so after we finished the calculation of like"}, {"start": 1692.48, "end": 1696.0400000000002, "text": " precision and recall metrics for every single query vector now this this"}, {"start": 1696.0400000000002, "end": 1700.64, "text": " function kind of populates this simple this is a simple like Python list it"}, {"start": 1700.64, "end": 1705.8000000000002, "text": " populates it with HTML syntax so we have as you see here here a table row table"}, {"start": 1705.8000000000002, "end": 1710.44, "text": " cell and we paste the like original query image and then we have we trade"}, {"start": 1710.44, "end": 1713.88, "text": " over all of the matches over the 50 matches we had all of those matches in"}, {"start": 1713.88, "end": 1718.0400000000002, "text": " the same row and then we close the cell we close the row and we repeat that for"}, {"start": 1718.0400000000002, "end": 1721.8400000000001, "text": " every single query vector that's 128 query vectors and that's how this thing"}, {"start": 1721.8400000000001, "end": 1727.2, "text": " is formed and after that let me kind of like jump into write HTML function I'm"}, {"start": 1727.2, "end": 1732.52, "text": " gonna put a breakpoint here and gonna press f5 so here we are it's gonna open"}, {"start": 1732.52, "end": 1737.48, "text": " up this demo HTML which is a template like HTML file and it's gonna replace"}, {"start": 1737.48, "end": 1742.52, "text": " some of the placeholder variables like as you can see here result we're gonna"}, {"start": 1742.52, "end": 1747.4, "text": " put this result HTML so that's the same just as Python string we just created"}, {"start": 1747.4, "end": 1751.32, "text": " that contains the content of the HTML file so yeah we're gonna replace all of"}, {"start": 1751.32, "end": 1757.04, "text": " these templates placeholders and finally we're gonna get let me jump to this web"}, {"start": 1757.04, "end": 1760.6, "text": " browser open it's gonna fail because I'm in WSL but that's just on my system it's"}, {"start": 1760.6, "end": 1764.8, "text": " gonna work for you much better and I'll just manually open it up so that's it"}, {"start": 1764.8, "end": 1769.24, "text": " when I press f5 this is the end of the program and I'm gonna open up the HTML"}, {"start": 1769.24, "end": 1773.36, "text": " okay here is the HTML let me open it up and here are the results and as you can"}, {"start": 1773.36, "end": 1777.44, "text": " see on the left hand side we've got query vectors on the right hand side we"}, {"start": 1777.44, "end": 1782.0, "text": " have the matches so the 50 matches for every single very vector and there is"}, {"start": 1782.0, "end": 1787.24, "text": " 128 of them and as you can see our results look pretty decent like if we"}, {"start": 1787.24, "end": 1791.56, "text": " focus on one particular image like I don't know like this shoe here we can"}, {"start": 1791.56, "end": 1794.8, "text": " see that most of these results are shoes they're not even though there are many"}, {"start": 1794.8, "end": 1798.88, "text": " other images in this data set like like bags or or whatever trousers I don't"}, {"start": 1798.88, "end": 1804.16, "text": " know there must be some like failure cases but I didn't manage to see anyone"}, {"start": 1804.16, "end": 1808.68, "text": " I don't see any failure case here in particular but yeah hopefully got some"}, {"start": 1808.68, "end": 1812.52, "text": " idea what Gina can do and let me just kind of mention that this second example"}, {"start": 1812.52, "end": 1817.64, "text": " with the QA the question-answer like chatbot is maybe a bit better because"}, {"start": 1817.64, "end": 1823.16, "text": " remember that Gina is a cloud native neural search so cloud and this example"}, {"start": 1823.16, "end": 1828.48, "text": " here what it does is it's very similar but like it opens up a chatbot so and"}, {"start": 1828.48, "end": 1831.96, "text": " it creates a Python server and so what happens is there is a communication"}, {"start": 1831.96, "end": 1835.68, "text": " happening between a Gina Python server and this JavaScript code in the browser"}, {"start": 1835.68, "end": 1840.0, "text": " and so you basically input like a query like a text query inside of the browser"}, {"start": 1840.0, "end": 1845.0, "text": " it gets embedded the Gina does its magic of finding the similar the most similar"}, {"start": 1845.0, "end": 1850.24, "text": " match so the most similar response the textual response and it communicates it"}, {"start": 1850.24, "end": 1856.3200000000002, "text": " back via the like this HTTPS I guess like a link to the JavaScript code and it"}, {"start": 1856.3200000000002, "end": 1860.44, "text": " just renders the the response it's a very similar example but it basically"}, {"start": 1860.44, "end": 1864.48, "text": " paints a clear picture the Gina is used for these cloud types of applications"}, {"start": 1864.48, "end": 1869.1200000000001, "text": " and because here we have explicitly like a Python server so hopefully found this"}, {"start": 1869.1200000000001, "end": 1872.48, "text": " video useful if you did share it out with a friend subscribe to the channel"}, {"start": 1872.48, "end": 1876.8, "text": " hit that bell icon also if you have any feedback on this video or what you would"}, {"start": 1876.8, "end": 1880.3600000000001, "text": " like me which ML tool you'd like me to cover next let me know down in the"}, {"start": 1880.36, "end": 1896.36, "text": " comments and also join the discord community until next time bye bye"}]