Semantic Segmentation in PyTorch | Neural Style Transfer #7
❤️ Become The AI Epiphany Patreon ❤️ ► https://www.patreon.com/theaiepiphany
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

In this video, I cover semantic segmentation - both basic theory and we also dig into the PyTorch implementation.

You'll learn about:
✔️ What is semantic segmentation
✔️ How to implement it in PyTorch using DeepLab V3
✔️ What are connected components and morph filters
✔️ How to post-process the raw model masks

Note: I'll cover the whole pipeline in the next video.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
✅ GitHub code: https://github.com/gordicaleksa/pytorch-naive-video-nst
✅ DeepLab V3 paper: https://arxiv.org/abs/1706.05587
✅ Morph filtering: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html
✅ Useful blog: https://www.learnopencv.com/pytorch-for-beginners-semantic-segmentation-using-torchvision/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

⌚️ Timetable: 
0:00 Semantic Segmentation (Basic Theory)
3:00 Semantic Segmentation (Code-Walkthrough)
8:25 Digital Image Processing (Basic Theory)
10:47 Mask post-processing (Code-Walkthrough)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany ► https://www.patreon.com/theaiepiphany
One-time donations: https://www.paypal.com/paypalme/theaiepiphany

Much love! ❤️

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 

💡 The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". 

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 
👋 CONNECT WITH ME ON SOCIAL 
LinkedIn ► https://www.linkedin.com/in/aleksagordic/ 
Twitter ► https://twitter.com/gordic_aleksa 

Instagram ► https://www.instagram.com/aiepiphany/ 
Facebook ► https://www.facebook.com/aiepiphany/ 

👨‍👩‍👧‍👦 JOIN OUR DISCORD COMMUNITY:
Discord ► https://discord.gg/peBrCpheKE

📢 SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack ► https://aiepiphany.substack.com/

💻 FOLLOW ME ON GITHUB FOR COOL PROJECTS: 
GitHub ► https://github.com/gordicaleksa 

📚 FOLLOW ME ON MEDIUM:
Medium ► https://gordicaleksa.medium.com/ 
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#semanticsegmentation #pytorch #neuralstyletransfer
 In this video we're going to learn about semantic segmentation and I'll first cover some basic theory and then we'll jump straight into coding in PyTorch, a project I recently open-sourced. So let's dig into it. So what exactly is semantic segmentation? So it's a computer vision task where the goal is to assign a single class to a single pixel. So you can see on the image there, certain pixels belong to the road, some pixels belong to the sidewalk, pedestrians, trees, etc. If you have all of this information, it's really easy to also do, for example, object detection because, for example, you just take the pedestrian, you just draw a bounding box around the pedestrian and you also know it's a pedestrian so you already did classification. The main metric for this task is something called mean intersectional reunion and it's pretty simple to understand. So basically you want to see the prediction your model makes, for example, for a single pedestrian and you want to see the intersection between the prediction and the ground truth, i.e. the true labels there. And the bigger the intersection, the better the metric and intuitively makes sense and you can see the nice image with the squares there. So it's used in all kinds of applications. The two main ones that fall into my mind are autonomous vehicles and the second one would be maybe mixed reality. I'm probably biased because I worked on HoloLens. The neural network that we'll be using in the code is DeepLab v3. So let me just briefly mention the DeepLab family. So it started, it was developed by Google and it started from v1 all the way to v3 plus. That's the the newest model but we are using v3 because it has official PyTorch implementation so it's really simple to use. On the screen on the top left you can see the input image and in the bottom right you can see the actual output I got with DeepLab v3. The model itself was trained on the Pascal VOC 2012 data set which has 21 classes. So 20 classes, foreground classes and one background class. And the classes compass, so I already mentioned background, we also have like person, airplane, bird etc. Because it was trained on 21 classes the output has 21 channels and the spatial resolution is the same as the input image. That's how semantic segmentation problem works. So how you get the most probable class for a single pixel is this. You just take, so you take a certain x y coordinate and you get 21 numbers right? Because you have 21 channels and wherever the highest number is, that's the highest probability, that's the class that's the most probable for that specific pixel. Say channel zero had the the biggest probability, that means background is the highest probability class for that specific pixel. It's as easy as that. The model you can see on the screen is actually FCN and not DeepLab, but the output shape is the thing I want you to see here. So it's 21 and the spatial resolution is the same as the input image as I already mentioned. Okay enough theory, let's jump into the code. This code is actually a whole neural style transfer video creation pipeline but we'll be focusing only on a single component and that's the segmentation. So let's see what the code looks like. So the extractPersonMaskFromFrames function basically takes an input frame and just extracts the pixels where the person is present. So line 55 we just like a regular pytorch thing, we just figure out whether the user has GPU or CPU. GPUs are always preferable when training, when using neural neural networks. Then we just instantiate DeepLab v3 model with the ResNet 101 backbone and we set pre-trained to true because we want to have a pre-trained model. Obviously we put it to the GPU if we have one and we set the model to the evaluation mode because certain layers like batch normalization and dropout will behave differently if we don't set this and we'll have some wrong outputs. Next up we create these transforms which will be applied to every single frame. So we want to maybe specify certain height and width because if we maybe our GPU doesn't have enough VRAM and it will just give you CUDA out of memory exception if we have a too big of a frame. So then we convert it to tensor to pytorch tensor and we do normalization using ImageNet statistics and this is just because of the way that the pytorch models were trained we have to do this processing step. Then we just create a image folder out of the frames that we want to process also standard pytorch think and we create a data loader and set the batch size to for example four or something because we want to use the parallel processing power of GPUs. Next steps are after figuring out whether the output directories are empty or full. This is like a cache mechanism if they already have some frames we want to just skip this state and this stage in the pipeline but if they are empty we want to proceed and we wrap all of this into this torch no-grat context because we are doing inference and otherwise pytorch will create computational graphs by default which will allocate lots of memory and eat up your VRAM so you want to you want to do this step it's really important. Next important thing is we just iterate through this data loader and we get image batches and we just this line 84 we just place the batch into the GPU because the model is also in GPU so you want to have tensors as well as the model on the same device otherwise you'll get some error and we just do the inference here we pass the image batch into the segmentation deep lab v3 model and because the output is actually order dictionary this is just like a thing you got to do you just extract the actual output using this out and then we place that resulting batch to the CPU and convert it to NumPy. Afterwards we iterate through the result batch which as I already mentioned contains so it's a dimension is n where the n is the number of like the batch size then 21 because we have 21 output channels and then height and width the same as the input frames. The out CPU is the has the shape I mentioned so 21 channels and height and width as the input frame and this is the main step so we do the arg max so that actually finds the thing I mentioned in the theoretical part so we just find the channel where we have the highest probability for that specific pixel and then the this equal equal person channel index will figure out the pixels where the person class was the highest probable one on the image so we just get a mask doing this we get a person mask i.e. we get the we'll have like a boolean value true on those pixels where the person was present as simple as that and then just some bureaucracy here times 255 will convert booleans into 0 to 255 binary image and converting into just explicitly converting it here to NumPy unsigned integer 8 type. After this we just do this post-processing step but before we dig into that part of the code I want to briefly also cover some theory behind the heuristics that will be used in that specific function. What it will do is it will just clean up some some components that the model spuriously outputted which are erroneous so we just want to clean the do some post-processing and it's pretty common in computer vision you usually have these hybrid approaches where the deep learning pipeline produces something and you just want to do some cleaning afterwards. So there are two things you want to know here the first one is connected components algorithm and the second one is the morphological filtering operations. So connected components are pretty simple we as humans can easily tell that the square is not connected to the circle i.e. there is there doesn't exist some like some some path of white pixels that's connecting them and what the algorithm should do here is just assign a different label to every one of these components like label 0 for background label 1 for square label 2 for circle having that information we can easily extract the circle or whatever component that we want and the colored image on the right just visualizes the thing i mentioned so it just visualizes the labels. So the second thing you need to know about is morphological filtering and you basically take the binary images the input and you just process it with something called the structuring element or the kernel which is also binary a simple binary mask and you can either do erosion where you get like the smaller area like you can see the j letter got smaller there or you can do something called dilation where you get the like bigger area and the way you implement this is if you know something about logic gates this is pretty much a multiple input and for the erosion or multiple input or gate for the dilation pretty simple. So finally opening something we'll be using actually and that's just a combination just a sequential you just sequentially apply first erosion and then dilation and it makes sense because if you see the input image before doing the opening operation you have those small dots after doing erosion they will disappear and after doing dilation you'll just you'll just be left with the j letter which will get to its previous size initial size. So on this slide you can see a concrete example of the person mask i got using deep lab wii 3 model and you can see by doing the opening we'll just kind of split that a small component that shouldn't be shouldn't supposed to be there and then after figuring out connected components and finding the second biggest one the first being background the second being person will be able to keep only the the person pixels simple as that. Back to the code we can now figure out what the post-processing method actually does and if we go jump here you can see that on line 26 we just create a kernel so that's the structuring element i mentioned in the morphological filtering slide and it's just a simple square of ones and after applying the using open cd's morphological function we just apply to the mask and we get the this thing called open mask which is the the initial mask from the deep lab model after applying opening operation. So then we just do connected components on the open mask and we get the labeled image out. Now given that labeled image from the connected components algorithm we need to figure out which component belongs to the background. So we take the upper part of the labeled image so the first 10 percent of the of the image and just count and see what's the most frequent value in that space. I call that discriminant subspace and the most common value is something we assume to be the background component background label and we get the background index here on line 37. The next step is to create a list of tuples where each tuple contains connected component components label and area and after that's line 43 and after sorting those according to the size of those areas and just filtering out the the the background label that we found above using the discriminant subspace we are left off with the the the first biggest component after the background that we just we assume to be person and we just grab the the using the so this this here it takes the the biggest leftover component and the zero just grabs the label so I'm left with person index and after after just checking which pixels contain exactly this label I'm left only with person pixels. It's pretty simple. If I go ahead and only visualize the mass that came out directly from the model we get a result like this and you can notice certain components here which do not belong to the person mask and which should be removed obviously so if I just go inside the post-processing method and after doing the morphological operations we get this and you can see on the right that the erosion already fixed this concrete mask but in some other cases we'll need connected components to remove the other connected components which do not belong to the person. So that covers the semantic segmentation theory and code. Hopefully you found this video useful and I hope you found this video useful. If you did consider subscribing and consider sharing this video and see you next time.
[{"start": 0.0, "end": 4.4, "text": " In this video we're going to learn about semantic segmentation and I'll first cover some basic theory"}, {"start": 4.4, "end": 8.96, "text": " and then we'll jump straight into coding in PyTorch, a project I recently open-sourced."}, {"start": 8.96, "end": 14.24, "text": " So let's dig into it. So what exactly is semantic segmentation? So it's a computer vision task where"}, {"start": 14.24, "end": 20.16, "text": " the goal is to assign a single class to a single pixel. So you can see on the image there, certain"}, {"start": 20.16, "end": 26.0, "text": " pixels belong to the road, some pixels belong to the sidewalk, pedestrians, trees, etc. If you have"}, {"start": 26.0, "end": 32.0, "text": " all of this information, it's really easy to also do, for example, object detection because, for"}, {"start": 32.0, "end": 37.04, "text": " example, you just take the pedestrian, you just draw a bounding box around the pedestrian and you"}, {"start": 37.04, "end": 42.8, "text": " also know it's a pedestrian so you already did classification. The main metric for this task is"}, {"start": 42.8, "end": 48.64, "text": " something called mean intersectional reunion and it's pretty simple to understand. So basically"}, {"start": 49.2, "end": 54.8, "text": " you want to see the prediction your model makes, for example, for a single pedestrian and you want"}, {"start": 54.8, "end": 62.72, "text": " to see the intersection between the prediction and the ground truth, i.e. the true labels there."}, {"start": 62.72, "end": 68.32, "text": " And the bigger the intersection, the better the metric and intuitively makes sense and you can"}, {"start": 68.32, "end": 73.6, "text": " see the nice image with the squares there. So it's used in all kinds of applications. The two main"}, {"start": 73.6, "end": 78.72, "text": " ones that fall into my mind are autonomous vehicles and the second one would be maybe"}, {"start": 78.72, "end": 83.6, "text": " mixed reality. I'm probably biased because I worked on HoloLens. The neural network that we'll be using"}, {"start": 83.6, "end": 88.72, "text": " in the code is DeepLab v3. So let me just briefly mention the DeepLab family. So it started, it was"}, {"start": 88.72, "end": 94.64, "text": " developed by Google and it started from v1 all the way to v3 plus. That's the the newest model but we"}, {"start": 94.64, "end": 99.52, "text": " are using v3 because it has official PyTorch implementation so it's really simple to use."}, {"start": 99.52, "end": 103.75999999999999, "text": " On the screen on the top left you can see the input image and in the bottom right you can see"}, {"start": 103.75999999999999, "end": 110.72, "text": " the actual output I got with DeepLab v3. The model itself was trained on the Pascal VOC 2012 data"}, {"start": 110.72, "end": 119.28, "text": " set which has 21 classes. So 20 classes, foreground classes and one background class. And the classes"}, {"start": 119.28, "end": 126.96000000000001, "text": " compass, so I already mentioned background, we also have like person, airplane, bird etc. Because it"}, {"start": 126.96000000000001, "end": 135.2, "text": " was trained on 21 classes the output has 21 channels and the spatial resolution is the same as the input"}, {"start": 135.2, "end": 141.51999999999998, "text": " image. That's how semantic segmentation problem works. So how you get the most probable class for"}, {"start": 141.51999999999998, "end": 148.16, "text": " a single pixel is this. You just take, so you take a certain x y coordinate and you get 21 numbers"}, {"start": 148.16, "end": 153.92, "text": " right? Because you have 21 channels and wherever the highest number is, that's the highest probability,"}, {"start": 154.72, "end": 161.76, "text": " that's the class that's the most probable for that specific pixel. Say channel zero had the"}, {"start": 161.76, "end": 167.84, "text": " the biggest probability, that means background is the highest probability class for that specific"}, {"start": 167.84, "end": 173.76, "text": " pixel. It's as easy as that. The model you can see on the screen is actually FCN and not DeepLab,"}, {"start": 173.76, "end": 178.64, "text": " but the output shape is the thing I want you to see here. So it's 21 and the spatial resolution"}, {"start": 178.64, "end": 182.23999999999998, "text": " is the same as the input image as I already mentioned. Okay enough theory, let's jump into"}, {"start": 182.23999999999998, "end": 189.12, "text": " the code. This code is actually a whole neural style transfer video creation pipeline but we'll"}, {"start": 189.12, "end": 194.72, "text": " be focusing only on a single component and that's the segmentation. So let's see what the code looks"}, {"start": 194.72, "end": 201.84, "text": " like. So the extractPersonMaskFromFrames function basically takes an input frame and just extracts"}, {"start": 203.12, "end": 210.16, "text": " the pixels where the person is present. So line 55 we just like a regular pytorch thing, we just"}, {"start": 210.16, "end": 216.72, "text": " figure out whether the user has GPU or CPU. GPUs are always preferable when training, when using"}, {"start": 216.72, "end": 225.92, "text": " neural neural networks. Then we just instantiate DeepLab v3 model with the ResNet 101"}, {"start": 226.56, "end": 233.28, "text": " backbone and we set pre-trained to true because we want to have a pre-trained model. Obviously"}, {"start": 233.28, "end": 239.92, "text": " we put it to the GPU if we have one and we set the model to the evaluation mode because"}, {"start": 239.92, "end": 246.32, "text": " certain layers like batch normalization and dropout will behave differently if we don't set this and"}, {"start": 246.32, "end": 252.07999999999998, "text": " we'll have some wrong outputs. Next up we create these transforms which will be applied to every"}, {"start": 252.07999999999998, "end": 260.56, "text": " single frame. So we want to maybe specify certain height and width because if we maybe our GPU"}, {"start": 260.56, "end": 266.15999999999997, "text": " doesn't have enough VRAM and it will just give you CUDA out of memory exception if we have a"}, {"start": 266.16, "end": 273.20000000000005, "text": " too big of a frame. So then we convert it to tensor to pytorch tensor and we do normalization"}, {"start": 273.20000000000005, "end": 279.76000000000005, "text": " using ImageNet statistics and this is just because of the way that the pytorch models were trained"}, {"start": 279.76000000000005, "end": 288.96000000000004, "text": " we have to do this processing step. Then we just create a image folder out of the frames that we"}, {"start": 288.96, "end": 296.71999999999997, "text": " want to process also standard pytorch think and we create a data loader and set the batch size to"}, {"start": 296.71999999999997, "end": 301.91999999999996, "text": " for example four or something because we want to use the parallel processing power of GPUs."}, {"start": 303.84, "end": 311.52, "text": " Next steps are after figuring out whether the output directories are empty or full. This is"}, {"start": 311.52, "end": 317.76, "text": " like a cache mechanism if they already have some frames we want to just skip this state and"}, {"start": 317.76, "end": 324.96, "text": " this stage in the pipeline but if they are empty we want to proceed and we wrap all of this into"}, {"start": 324.96, "end": 331.59999999999997, "text": " this torch no-grat context because we are doing inference and otherwise pytorch will create"}, {"start": 331.59999999999997, "end": 337.59999999999997, "text": " computational graphs by default which will allocate lots of memory and eat up your VRAM"}, {"start": 337.59999999999997, "end": 341.84, "text": " so you want to you want to do this step it's really important. Next important thing is we"}, {"start": 341.84, "end": 351.44, "text": " just iterate through this data loader and we get image batches and we just this line 84"}, {"start": 351.44, "end": 356.79999999999995, "text": " we just place the batch into the GPU because the model is also in GPU so you want to have tensors"}, {"start": 356.79999999999995, "end": 363.2, "text": " as well as the model on the same device otherwise you'll get some error and we just do the"}, {"start": 363.2, "end": 370.79999999999995, "text": " inference here we pass the image batch into the segmentation deep lab v3 model and because the"}, {"start": 370.8, "end": 378.32, "text": " output is actually order dictionary this is just like a thing you got to do you just extract the"}, {"start": 378.32, "end": 385.84000000000003, "text": " actual output using this out and then we place that resulting batch to the CPU and convert it to"}, {"start": 385.84000000000003, "end": 394.32, "text": " NumPy. Afterwards we iterate through the result batch which as I already mentioned contains so"}, {"start": 394.32, "end": 401.59999999999997, "text": " it's a dimension is n where the n is the number of like the batch size then 21 because we have"}, {"start": 401.59999999999997, "end": 408.96, "text": " 21 output channels and then height and width the same as the input frames. The out CPU is the"}, {"start": 410.15999999999997, "end": 416.8, "text": " has the shape I mentioned so 21 channels and height and width as the input frame and this is"}, {"start": 416.8, "end": 422.15999999999997, "text": " the main step so we do the arg max so that actually finds the thing I mentioned in the theoretical"}, {"start": 422.16, "end": 428.64000000000004, "text": " part so we just find the channel where we have the highest probability for that specific pixel"}, {"start": 428.64000000000004, "end": 435.68, "text": " and then the this equal equal person channel index will figure out the pixels where the person class"}, {"start": 435.68, "end": 442.56, "text": " was the highest probable one on the image so we just get a mask doing this we get a person mask i.e."}, {"start": 442.56, "end": 448.88, "text": " we get the we'll have like a boolean value true on those pixels where the person was present"}, {"start": 448.88, "end": 458.71999999999997, "text": " as simple as that and then just some bureaucracy here times 255 will convert booleans into 0 to 255"}, {"start": 458.71999999999997, "end": 466.71999999999997, "text": " binary image and converting into just explicitly converting it here to NumPy unsigned integer 8"}, {"start": 467.36, "end": 475.28, "text": " type. After this we just do this post-processing step but before we dig into that part of the code"}, {"start": 475.28, "end": 482.32, "text": " I want to briefly also cover some theory behind the heuristics that will be used in that specific"}, {"start": 482.32, "end": 489.76, "text": " function. What it will do is it will just clean up some some components that the model spuriously"}, {"start": 489.76, "end": 497.2, "text": " outputted which are erroneous so we just want to clean the do some post-processing and it's"}, {"start": 497.2, "end": 501.91999999999996, "text": " pretty common in computer vision you usually have these hybrid approaches where the deep learning"}, {"start": 501.92, "end": 506.08000000000004, "text": " pipeline produces something and you just want to do some cleaning afterwards. So there are two things"}, {"start": 506.08000000000004, "end": 510.40000000000003, "text": " you want to know here the first one is connected components algorithm and the second one is the"}, {"start": 511.12, "end": 516.8000000000001, "text": " morphological filtering operations. So connected components are pretty simple we as humans can"}, {"start": 516.8000000000001, "end": 523.6, "text": " easily tell that the square is not connected to the circle i.e. there is there doesn't exist some"}, {"start": 524.16, "end": 529.28, "text": " like some some path of white pixels that's connecting them and what the algorithm should"}, {"start": 529.28, "end": 537.52, "text": " do here is just assign a different label to every one of these components like label 0 for background"}, {"start": 537.52, "end": 543.12, "text": " label 1 for square label 2 for circle having that information we can easily extract the circle"}, {"start": 543.12, "end": 548.0799999999999, "text": " or whatever component that we want and the colored image on the right just visualizes the thing i"}, {"start": 548.0799999999999, "end": 552.56, "text": " mentioned so it just visualizes the labels. So the second thing you need to know about is morphological"}, {"start": 552.56, "end": 558.88, "text": " filtering and you basically take the binary images the input and you just process it with something"}, {"start": 558.88, "end": 564.64, "text": " called the structuring element or the kernel which is also binary a simple binary mask and you can"}, {"start": 564.64, "end": 570.32, "text": " either do erosion where you get like the smaller area like you can see the j letter got smaller"}, {"start": 570.32, "end": 577.68, "text": " there or you can do something called dilation where you get the like bigger area and the way"}, {"start": 577.68, "end": 582.64, "text": " you implement this is if you know something about logic gates this is pretty much a multiple"}, {"start": 582.64, "end": 592.24, "text": " input and for the erosion or multiple input or gate for the dilation pretty simple. So finally"}, {"start": 592.24, "end": 597.68, "text": " opening something we'll be using actually and that's just a combination just a sequential"}, {"start": 597.68, "end": 604.24, "text": " you just sequentially apply first erosion and then dilation and it makes sense because if you see the"}, {"start": 604.24, "end": 610.56, "text": " input image before doing the opening operation you have those small dots after doing erosion"}, {"start": 610.56, "end": 616.88, "text": " they will disappear and after doing dilation you'll just you'll just be left with the j letter which"}, {"start": 616.88, "end": 622.7199999999999, "text": " will get to its previous size initial size. So on this slide you can see a concrete example of the"}, {"start": 622.7199999999999, "end": 628.64, "text": " person mask i got using deep lab wii 3 model and you can see by doing the opening we'll just kind"}, {"start": 628.64, "end": 635.8399999999999, "text": " of split that a small component that shouldn't be shouldn't supposed to be there and then after"}, {"start": 635.84, "end": 641.12, "text": " figuring out connected components and finding the second biggest one the first being background the"}, {"start": 641.12, "end": 648.08, "text": " second being person will be able to keep only the the person pixels simple as that. Back to the code"}, {"start": 648.08, "end": 653.9200000000001, "text": " we can now figure out what the post-processing method actually does and if we go jump here"}, {"start": 654.8000000000001, "end": 660.1600000000001, "text": " you can see that on line 26 we just create a kernel so that's the structuring element i mentioned"}, {"start": 660.16, "end": 668.16, "text": " in the morphological filtering slide and it's just a simple square of ones and after applying the"}, {"start": 668.16, "end": 677.68, "text": " using open cd's morphological function we just apply to the mask and we get the this thing called"}, {"start": 677.68, "end": 684.88, "text": " open mask which is the the initial mask from the deep lab model after applying opening operation."}, {"start": 684.88, "end": 692.16, "text": " So then we just do connected components on the open mask and we get the labeled image out. Now"}, {"start": 692.16, "end": 697.28, "text": " given that labeled image from the connected components algorithm we need to figure out which"}, {"start": 697.28, "end": 704.08, "text": " component belongs to the background. So we take the upper part of the labeled image so the first 10"}, {"start": 704.08, "end": 711.52, "text": " percent of the of the image and just count and see what's the most frequent value in that space. I"}, {"start": 711.52, "end": 717.92, "text": " call that discriminant subspace and the most common value is something we assume to be the"}, {"start": 717.92, "end": 724.96, "text": " background component background label and we get the background index here on line 37. The next"}, {"start": 724.96, "end": 732.56, "text": " step is to create a list of tuples where each tuple contains connected component components label"}, {"start": 732.56, "end": 741.28, "text": " and area and after that's line 43 and after sorting those according to the size of those areas and"}, {"start": 741.28, "end": 748.0799999999999, "text": " just filtering out the the the background label that we found above using the discriminant subspace"}, {"start": 748.0799999999999, "end": 755.68, "text": " we are left off with the the the first biggest component after the background that we just"}, {"start": 755.68, "end": 763.68, "text": " we assume to be person and we just grab the the using the so this this here it takes the the"}, {"start": 763.68, "end": 772.64, "text": " biggest leftover component and the zero just grabs the label so I'm left with person index and after"}, {"start": 773.52, "end": 780.56, "text": " after just checking which pixels contain exactly this label I'm left only with person pixels."}, {"start": 780.56, "end": 786.2399999999999, "text": " It's pretty simple. If I go ahead and only visualize the mass that came out directly from the model"}, {"start": 786.2399999999999, "end": 795.8399999999999, "text": " we get a result like this and you can notice certain components here which do not belong to the person"}, {"start": 797.28, "end": 804.4799999999999, "text": " mask and which should be removed obviously so if I just go inside the post-processing method"}, {"start": 804.48, "end": 812.72, "text": " and after doing the morphological operations we get this and you can see on the right that"}, {"start": 812.72, "end": 820.4, "text": " the erosion already fixed this concrete mask but in some other cases we'll need connected components"}, {"start": 820.4, "end": 826.64, "text": " to remove the other connected components which do not belong to the person. So that covers the"}, {"start": 826.64, "end": 833.04, "text": " semantic segmentation theory and code. Hopefully you found this video useful and I hope you found"}, {"start": 833.04, "end": 842.64, "text": " this video useful. If you did consider subscribing and consider sharing this video and see you next time."}]