  It's a meaningful and attractive topic to build a general and inclusive
segmentation model that can recognize more categories in various scenarios. A
straightforward way is to combine the existing fragmented segmentation datasets
and train a multi-dataset network. However, there are two major issues with
multi-dataset segmentation: (1) the inconsistent taxonomy demands manual
reconciliation to construct a unified taxonomy; (2) the inflexible one-hot
common taxonomy causes time-consuming model retraining and defective
supervision of unlabeled categories. In this paper, we investigate the
multi-dataset segmentation and propose a scalable Language-guided Multi-dataset
Segmentation framework, dubbed LMSeg, which supports both semantic and panoptic
segmentation. Specifically, we introduce a pre-trained text encoder to map the
category names to a text embedding space as a unified taxonomy, instead of
using inflexible one-hot label. The model dynamically aligns the segment
queries with the category embeddings. Instead of relabeling each dataset with
the unified taxonomy, a category-guided decoding module is designed to
dynamically guide predictions to each datasets taxonomy. Furthermore, we adopt
a dataset-aware augmentation strategy that assigns each dataset a specific
image augmentation pipeline, which can suit the properties of images from
different datasets. Extensive experiments demonstrate that our method achieves
significant improvements on four semantic and three panoptic segmentation
datasets, and the ablation study evaluates the effectiveness of each component.
