  This paper is written because I receive several inquiry emails saying it is
hard to achieve good results when applying token repetition learning
techniques. If REP (proposed by me) or Pointer-Mixture (proposed by Jian Li) is
directly applied to source code to decide all token repetitions, the model
performance will decrease sharply. As we use pre-order traversal to traverse
the Abstract Syntax Tree (AST) to generate token sequence, tokens corresponding
to AST grammar are ignored when learning token repetition. For non-grammar
tokens, there are many kinds: strings, chars, numbers and identifiers. For each
kind of tokens, we try to learn its repetition pattern and find that only
identifiers have the property of token repetition. For identifiers, there are
also many kinds such as variables, package names, method names, simple types,
qualified types or qualified names. Actually, some kinds of identifiers such as
package names, method names, qualified names or qualified types are unlikely to
be repeated. Thus, we ignore these kinds of identifiers that are unlikely to be
repeated when learning token repetition. This step is crucial and this
important implementation trick is not clearly presented in the paper because we
think it is trivial and too many details may bother readers. We offer the
GitHub address of our model in our conference paper and readers can check the
description and implementation in that repository. Thus, in this paper, we
supplement the important implementation optimization details for the already
published papers.
