  UNet [27] is widely used in semantic segmentation due to its simplicity and
effectiveness. However, its manually-designed architecture is applied to a
large number of problem settings, either with no architecture optimizations, or
with manual tuning, which is time consuming and can be sub-optimal. In this
work, firstly, we propose Markov Random Field Neural Architecture Search
(MRF-NAS) that extends and improves the recent Adaptive and Optimal Network
Width Search (AOWS) method [4] with (i) a more general MRF framework (ii)
diverse M-best loopy inference (iii) differentiable parameter learning. This
provides the necessary NAS framework to efficiently explore network
architectures that induce loopy inference graphs, including loops that arise
from skip connections. With UNet as the backbone, we find an architecture,
MRF-UNet, that shows several interesting characteristics. Secondly, through the
lens of these characteristics, we identify the sub-optimality of the original
UNet architecture and further improve our results with MRF-UNetV2. Experiments
show that our MRF-UNets significantly outperform several benchmarks on three
aerial image datasets and two medical image datasets while maintaining low
computational costs. The code is available at:
https://github.com/zifuwanggg/MRF-UNets.
