In order to familiarize myself with semantic segmentation and convolutional neural networks I am going through this tutorial by MathWorks:

Semantic Segmentation Using Deep Learning

I did not use the pretrained version of Segnet since I wanted to test on my custom data set. All code is the same, however I have different classes, and fewer labels. Below image shows the label name and amount of pixels associated with each.



To make up for the low pixel data for class 2, median frequency balancing was performed.

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount
classWeights = median(imageFreq) ./ imageFreq


I proceed to train the network using the code provided in the example with the options and lgraph unchanged. The  SegNet network is created with weights initialized from the VGG-16 network.

Unlike the example, I get a much lower global accuracy:



To gain further insight I plotted the Mini-batch accuracy and Mini-batch loss against each iteration. 



It is clearly seen that the accuracy fluctuates wildly and ends up worse than it started, so the network learned absolutely nothing! However the loss decreased gradually.  

A possible solution I propose would be to use inverse frequency balancing. However, in the example above, median frequency balancing was already performed, so I doubt how much this would help.

Is the terrible performance related to simply not having enough training data? Can anything be be done to improve performance with existing data?

Any suggestions are greatly appreciated.

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]).

Formula for categorical crossentropy (S - samples, C - classess, $s \in c $ - sample belongs to class c) is:

$$ -\frac{1}{N} \sum_{s\in S} \sum_{c \in C} 1_{s\in c} log {p(s \in c)} $$ 

For case when classes are exclusive, you don't need to sum over them - for each sample only non-zero value is just $-log p(s \in c)$ for true class c.

This allows to conserve time and memory. Consider case of 10000 classes when they are mutually exclusive - just 1 log instead of summing up 10000 for each sample, just one integer instead of 10000 floats.

Formula is the same in both cases, so no impact on accuracy should be there. 
