  Deep learning has thrived by training on large-scale datasets. However, in
robotics applications sample efficiency is critical. We propose a novel
adaptive masked proxies method that constructs the final segmentation layer
weights from few labelled samples. It utilizes multi-resolution average pooling
on base embeddings masked with the label to act as a positive proxy for the new
class, while fusing it with the previously learned class signatures. Our method
is evaluated on PASCAL-$5^i$ dataset and outperforms the state-of-the-art in
the few-shot semantic segmentation. Unlike previous methods, our approach does
not require a second branch to estimate parameters or prototypes, which enables
it to be used with 2-stream motion and appearance based segmentation networks.
We further propose a novel setup for evaluating continual learning of object
segmentation which we name incremental PASCAL (iPASCAL) where our method
outperforms the baseline method. Our code is publicly available at
https://github.com/MSiam/AdaptiveMaskedProxies.
