  Offloading computing to edge servers is a promising solution to support
growing video understanding applications at resource-constrained IoT devices.
Recent efforts have been made to enhance the scalability of such systems by
reducing inference costs on edge servers. However, existing research is not
directly applicable to pixel-level vision tasks such as video semantic
segmentation (VSS), partly due to the fluctuating VSS accuracy and segment
bitrate caused by the dynamic video content. In response, we present Penance, a
new edge inference cost reduction framework. By exploiting softmax outputs of
VSS models and the prediction mechanism of H.264/AVC codecs, Penance optimizes
model selection and compression settings to minimize the inference cost while
meeting the required accuracy within the available bandwidth constraints. We
implement Penance in a commercial IoT device with only CPUs. Experimental
results show that Penance consumes a negligible 6.8% more computation resources
than the optimal strategy while satisfying accuracy and bandwidth constraints
with a low failure rate.
