  In natural language understanding (NLU) production systems, users' evolving
needs necessitate the addition of new features over time, indexed by new
symbols added to the meaning representation space. This requires additional
training data and results in ever-growing datasets. We present the first
systematic investigation of this incremental symbol learning scenario. Our
analysis reveals a troubling quirk in building broad-coverage NLU systems: as
the training dataset grows, performance on the new symbol often decreases if we
do not accordingly increase its training data. This suggests that it becomes
more difficult to learn new symbols with a larger training dataset. We show
that this trend holds for multiple mainstream models on two common NLU tasks:
intent recognition and semantic parsing. Rejecting class imbalance as the sole
culprit, we reveal that the trend is closely associated with an effect we call
source signal dilution, where strong lexical cues for the new symbol become
diluted as the training dataset grows. Selectively dropping training examples
to prevent dilution often reverses the trend, showing the over-reliance of
mainstream neural NLU models on simple lexical cues. Code, models, and data are
available at https://aka.ms/nlu-incremental-symbol-learning
