  Word-level sign language recognition (WSLR) is a fundamental task in sign
language interpretation. It requires models to recognize isolated sign words
from videos. However, annotating WSLR data needs expert knowledge, thus
limiting WSLR dataset acquisition. On the contrary, there are abundant
subtitled sign news videos on the internet. Since these videos have no
word-level annotation and exhibit a large domain gap from isolated signs, they
cannot be directly used for training WSLR models. We observe that despite the
existence of a large domain gap, isolated and news signs share the same visual
concepts, such as hand gestures and body movements. Motivated by this
observation, we propose a novel method that learns domain-invariant visual
concepts and fertilizes WSLR models by transferring knowledge of subtitled news
sign to them. To this end, we extract news signs using a base WSLR model, and
then design a classifier jointly trained on news and isolated signs to coarsely
align these two domain features. In order to learn domain-invariant features
within each class and suppress domain-specific features, our method further
resorts to an external memory to store the class centroids of the aligned news
signs. We then design a temporal attention based on the learnt descriptor to
improve recognition performance. Experimental results on standard WSLR datasets
show that our method outperforms previous state-of-the-art methods
significantly. We also demonstrate the effectiveness of our method on
automatically localizing signs from sign news, achieving 28.1 for<EMAIL_ADDRESS>