---
abstract: 'In recent years, deep learning models have shown great potential in source code modeling and analysis. Generally, deep learning-based approaches are problem-specific and data-hungry. A challenging issue of these approaches is that they require training from starch for a different related problem. In this work, we propose a transfer learning-based approach that significantly improves the performance of deep learning-based source code models. In contrast to traditional learning paradigms, transfer learning can transfer the knowledge learned in solving one problem into another related problem. First, we present two recurrent neural network-based models RNN and GRU for the purpose of transfer learning in the domain of source code modeling. Next, via transfer learning, these pre-trained (RNN and GRU) models are used as feature extractors. Then, these extracted features are combined into *attention* learner for different downstream tasks. The *attention* learner leverages from the learned knowledge of pre-trained models and fine-tunes them for a specific downstream task. We evaluate the performance of the proposed approach with extensive experiments with the source code suggestion task. The results indicate that the proposed approach outperforms the state-of-the-art models in terms of accuracy, precision, recall, and F-measure without training the models from scratch.'
address:
- 'College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics (NUAA), Nanjing 211106, China'
- 'Key Laboratory of Safety-Critical Software, NUAA, Ministry of Industry and Information Technology, Nanjing 211106, China'
- 'Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing 210093, China'
author:
- Yasir Hussain
- Zhiqiu Huang
- Yu Zhou
- Senzhang Wang
bibliography:
- 'menuscript.bib'
title: Deep Transfer Learning for Source Code Modeling
---

Transfer Learning, Deep Neural Language Models, Source Code Modeling, Attention Learning.

Introduction
============

Source code suggestion and syntax error fixing are vital features of a modern integrated development environment (IDE). These features help software developers to build and debug software rapidly. Recently, deep learning-based language models have shown great potential in various source code modeling tasks [@Allamanis2016; @Alon2019; @Santos2018; @Gupta2018; @Hussain2018; @Iyer2016; @Fowkes2017; @Raychev2013a; @Sethi2017; @White2015].

Some researchers have worked on source code suggestion [@Hussain2018; @White2015; @Raychev2013a] task in which they suggest the next possible source code token. They take a fixed size context prior to the prediction position as features and help the software developers by suggesting the next possible code token. Some researchers have worked on syntax error detection and correction [@Santos2018; @Gupta2018] problem. They consider the source code syntax as features and use them for the correction of the syntax errors found in a source code file. Several researchers have focused on the code summarization task [@Fowkes2017; @Iyer2016; @Allamanis2016] in which they help summarize the source code for better understand the working of it. Further, some works focus on method naming [@Alon2019] which gives a meaningful name to a source code method. Some works are focused on source code generation [@Sethi2017] task in which they use the natural language queries to help generate the source code.

A challenging issue of these approaches is that they are problem-specific which requires training from starch for a different related problem. Further, deep learning-based approaches are data-hungry which means they require training on large data set to produce satisfactory results. Furthermore, deep learning models requires days to train while training on a large dataset. To overcome these issues, we exploit the concept of transfer learning in this work. In transfer learning, the learned knowledge from a pre-trained model is extracted and then be used for a similar downstream task [@salem2019utilizing].

This work proposes a transfer learning-based approach that significantly improves the performance of deep learning-based source code models. First, we exploit the concept of transfer learning for deep learning-based source code language models. The key idea is to use a pre-trained source code language model to transfer the learned knowledge from it for a different related problem. We train two different variants of recurrent neural network-based models RNN and GRU for the purpose of transfer learning. Then, we combine the learned knowledge of pre-trained (RNN and GRU) models into *attention* learner for a downstream task. The *attention* learner leverage from the learned knowledge of pre-trained models and fine-tunes it for a specific downstream task. Via transfer learning, pre-trained models are used to extract generalized features and then fine-tune them for a target task without requiring the model training from scratch. We evaluate the proposed approach with the downstream task of source code suggestion.

Proposed approach {#Methodology}
=================

This section discusses the proposed framework in detail. The \[fig:framework\] shows the overall architecture of the proposed approach. In this section, we first discuss the pre-trained models for the purpose of transfer learning. Next, we discuss the extraction of knowledge from pre-trained models combined with *attention* learner for the downstream task of source code suggestion. The *attention* learner is used to fine-tune the models for the target task by paying attention to the features which are related to the target task. The details about each step are given in the following subsections.

![image](img/TransferLearning-Framework){width="\linewidth"}

Transfer Learning {#TransferLearning}
-----------------

In Transfer learning the knowledge learned in solving one problem is transferred and fine-tuned for another related problem. In recent years, transfer learning methods have been successfully applied in different fields such as metric learning [@hu2015], machine learning [@Duan2009], and dimensionality reduction [@pan2008]. Further, transfer learning has been extensively studied for various tasks in the field of image and text classification [@khan2019; @Shin2016; @Zhang2018; @Yuan2017; @KRAUS2017]. \[fig:learningComparision\] shows the difference between traditional learning and transfer learning-based approaches for source code modeling.

For the purpose of transfer learning, we first need a pre-trained model. There are several CNN (GoogLeNet [@Szegedy2015] , VGGNet [@Dan2018] and ResNet [@He2016]) and NLP (BERT [@Devlin2018], Transformer-XL [@Dai2019], OpenAIs GPT-2 [@radford2019]) based models for image and text classification respectively. The source code strictly follows the rules defined by their grammar[^1], thus these models are not suitable for our purpose. In this work, we first train two variants of recurrent neural networks-based models RNN and GRU for the purpose of transfer learning in the field of source code. We choose RNN [@White2015; @Raychev2013a] and GRU [@Hussain2018; @Gupta2018] because of their recent success in the modeling of source code. To train the models for transfer learning first, we gather the data set used in previous studies [@Hussain2018; @Hindle2012; @Nguyen2018]. \[Table:TopDataStatistics\] shows the details of the data set used to build pre-trained models. By combining all collected projects, we end up with 13 million code tokens with a large vocabulary of size 177,342.

   [Projects]{}   [Version]{}      [LOC]{}        [Total]{}      [Vocab Size]{}
  -------------- ------------- --------------- ---------------- ----------------
       ant          1.10.5         149,960         920,978           17,132
    cassandra       3.11.3         318,704         2734218           33,424
       db40           7.2          241,766        1,435,382          20,286
       jgit          5.1.3         199,505        1,538,905          20,970
       poi           4.0.0         387,203        2,876,253          47,756
      maven          3.6.0         69,840          494,379           8,066
      batik         1.10.0         195,652        1,246,157          21,964
       jts          1.16.0         91,387          611,392           11,903
      itext         5.5.13         161,185        1,164,362          19,113
      antlr          4.7.1         56,085          407,248           6,813
      Total                     [1,871,287]{}   [13,429,274]{}    [177,342]{}

  : Data set used to pre-train models for transfer learning. The table shows name of the project, version of the project, line of code (LOC), total code tokens and unique code tokens found in each project.[]{data-label="Table:TopDataStatistics"}

Pre-Training Models for Transfer Learning {#MethodologyTraining}
-----------------------------------------

All models are trained on Intel(R) Xeon(R) Silver 4110 CPU 2.10GHz x 32 cores and 128GB of ram running Ubuntu 18.04.2 LTS operating system, equipped with the latest NVIDIA GeForce RTX 2080. The \[Table:DeepModelsArch\] shows the architecture of trained models used for transfer learning. We follow the same approach used in previous works [@Hussain2018; @White2015; @Raychev2013a] to pre-process the data set. To build a global vocabulary system [@White2015], we remove all code tokens appearing less than three times in the collected data set which ends up with the vocabulary size of 88,013 unique code tokens. We map the vocabulary to a continuous feature vector of dense size *300* similar to Word2Vec [@Rong2014]. We use *300* hidden units with context size ($\tau$) of *20* as studied by White et al. [@White2015]. For each model training we employ *Adam* [@Kingma2014] optimizer with the default learn rate of *0.001*. To control overfitting, we use *Dropout* [@Gajbhiye2018]. Each model is trained until it converges by employing *early stop* [@Alon2019] with the patience of three consecutive hits on the validation loss. One important thing to point out here is that the training process of these models is one time and do not need retraining. The trained models are publicly available[^2] for the purpose of transfer learning.

[\*[5]{}[r]{}]{} & Type & Size & Activations\
Input & Code embedding & 300 & &\
Estimator & RNN,GRU & 300 & tanh &\
Over Fitting & Dropout & & &\
Output & Dense & $V$ & softmax &\
Loss & Categorical cross entropy& & &\
Optimizer & Adam & & &\

Learning to Transfer Knowledge
------------------------------

For transfer learning first, we prepared the pre-trained models as described earlier in this section. Then, we use these pre-trained models to transfer the learned knowledge for a downstream task. A key insight is to freeze the learned knowledge in pre-trained models to keep the learned knowledge unchanged and fine-tunes the learned knowledge for a target task. Recently, *attention*-based approaches have shown great potential in various fields such as speech recognition [@Chorowski2015], machine translation [@Luong2015; @bahdanau2016], and more [@Vaswani2017; @Alon2019]. In proposed approach, we use the *attention* learner to fine-tune the model for a target task. The *attention* learner pays attention to the task-specific features to achieve optimal performance. The \[fig:TransferLearnArch\] shows the architecture design of our proposed transfer learning-based *attention* model. We show the effectiveness of the proposed approach with the downstream task of source code suggestion. A source code suggestion engine recommends the next possible source code token given a context.

### PreProcessing {#PreProcessing}

This section briefly introduces each of the strategic steps that we apply for the task of source code suggestion. Following common practices[@White2015], we perform normalization, tokenization and feature extraction. For the illustrating example, \[Table:Preprocessing\] shows the effect of each preprocessing step. We discuss each step in detail in the following subsections.

[ C[3.5cm]{} | C[8cm]{} ]{} Original Source code & ![image](img/OrignalCode.png){width="0.7\linewidth"}\
\
Normalized Source Code & ![image](img/NormalizedCode.png){width="0.6\linewidth"}\
\
Tokenized Source Code & ![image](img/TokanizedCode.png){width="\linewidth"}\
\
Vectorized Source Code & ![image](img/VocabularyCode.png){width="0.7\linewidth"}\

### Normalization {#normalization .unnumbered}

One of the vital preprocessing steps is to normalize the data set. Usually, a data set contains some values which are unnecessary for a particular task, these type of values will intensely upset the outcome of the analysis. For this purpose, we normalize the source code files by removing all blank lines, inline and block-level comments. We replace all constant numerical values to their generic types (e.g. 1 = IntVal, 1.2 = FloatVal) and replace constant strings with a generic *StringVal* token.

### Tokenization {#tokenization .unnumbered}

After normalizing source code files, we tokenize the source code files. Tokenization is the process of extracting terms/words from the data set. For this purpose, each source code file is parsed into a sequence of space-separated code tokens. Each sequence is then parted into multiple subsequences of fixed size context *20* [@White2015].

### Feature Extraction {#feature-extraction .unnumbered}

To convert the source code sequences into a form that is suitable for training deep learning models we perform a series of transformations. First, we replace common tokens occurring only once in the corpus with a special token *unk* to build a global vocabulary system. Next, we build the vocabulary where each unique source code token corresponds to an entry in the vocabulary. Then each source code token is assigned a unique positive integer corresponding to its vocabulary index to convert the sequences (feature vectors) into a form that is suitable for training a deep learning model.

![image](img/attention-model){width="\linewidth"}

         Layers                Type              Size   Activations
  -- -------------- --------------------------- ------ -------------
         Input            Code embedding         300   
       Estimator              RNN,GRU            300       tanh
       Combining            Concatenate                
      Over Fitting            Dropout                  
       Attention         Attention Learner             
         Output                Dense             $V$      softmax
          Loss       Categorical cross entropy         
       Optimizer               Adam                    

Evaluation {#Evaluation}
==========

In this section, we evaluate the effectiveness of our proposed approach by investigating the following research questions:

-   RQ1: Does the proposed approach outperform the state-of-the-art approaches? if yes, to what extent?

-   RQ2: How well the proposed approach performs in terms of source code suggestion task as compared to other baseline approaches?

-   RQ3: Does normalization help to improve the performance of the proposed approach? If yes, to what extent?

To answer the research question (RQ1), we compare the performance of the proposed approach with the state-of-the-art approaches. To answer the research question (RQ2), we evaluate and compare the proposed approach for source code suggestion tasks with other baseline approaches. To answer the research question (RQ3), We conduct a comparative analysis to show the impact of normalization on model performance.

### Data set {#Dataset}

To empirically evaluate our work, we collected java projects from *GitHub* a well-known open-source software repositories provider. We gather the top five java projects sorted by the number of star from *GitHub* at the time of this study. We download the latest snapshot of the project usually named as the *master branch*. Here, we choose the projects which are not used while training the pre-trained models discussed in \[Methodology\]. The \[Table:DataSet\] shows the version of each project, the total number of code lines, total code tokens and unique code tokens found in each project.

[\*[6]{}[r]{}]{} & & &\
[Projects]{} & [Version]{} &[LOC]{} &[Total]{} & [Vocab Size ($V$)]{}\
elastic-search & v7.0.0 & 210,357 & 1,765,479 & 24,691\
java-design-patterns & v1.20.0 & 30,784 & 200,344 & 5,649\
RxJava & v2.2.8 & 257,704 & 1,908,258 & 12,230\
interviews & v1.0 & 13,750 & 80,074 & 1,157\
spring-boot & v2.2.0.M2 & 224,465 & 1,813,891 & 34,609\

Process and Metrics
-------------------

### Process: {#process .unnumbered}

We train several baseline models for the evaluation of this work. The proposed approach is evaluated in the following manner;

-   We train RNN [@Raychev2013a] based model as baseline similar to White et al. [@White2015].

-   We train GRU based deep neural model as baseline similar to Cho et al.[@Cho2014]

-   We train transfer learning-based *attention* model by following the proposed approach as discussed in \[Methodology\].

We choose the approach proposed by White et al. [@White2015] for comparison because they have shown the effectiveness of their approach with the similar task of source code suggestion and as far as we know, considered as the state-of-the-art approach. We train the GRU [@Cho2014] based model as the baseline because GRU based model is an advanced version of RNN which removes the vanishing gradient problem and performs better[@Hussain2018]. To empirically evaluate our work, we repeat our experiment on each project separately. We randomly partition the projects into ten equal lines of code folds from which one fold is used for testing, one fold is used for model parameter optimization (validation) and rest of the folds are used for model training. The \[Table:DeepAttantionModelsArch\] shows the proposed transfer learning-based attention model architecture. First, we preprocess the data set as discussed earlier in \[PreProcessing\]. Then, we map the vocabulary to a continuous feature vector of dense size *300* similar to Word2Vec [@Rong2014]. We use *300* hidden units with context size ($\tau$) of *20* as studied by White et al. [@White2015]. For each model training we employ *Adam* [@Kingma2014] optimizer with the default learn rate of *0.001*. To control overfitting, we use *Dropout* [@Gajbhiye2018]. Each model is trained until it converges by employing *early stop* [@Alon2019] with the patience of three consecutive hits on the validation loss.

### Metrics: {#metrics .unnumbered}

For the evaluation of the proposed approach, we choose similar metrics as in previous studies. We choose top-k accuracy [@White2015; @Raychev2013a] and Mean Reciprocal Rank (MRR) [@Hussain2018; @Nguyen2018; @Santos2018] metrics for the evaluation of this work. Further, to evaluate the performance of the proposed approach we measure the precision, recall and F-measure scores which are widely used metrics [@Alon2019]. Furthermore, to evaluate the significance of the proposed approach we perform ANOVA statistical testing.

$$\label{eq:accuracy}
Accuracy = \frac{TP+TN}{TP+FN+FP+TN}$$

$$\label{eq:precision}
Precision = \frac{TP}{TP+FP}$$

$$\label{eq:recall}
Recall = \frac{TP}{TP+FN}$$

$$\label{eq:fmeaure}
\text{F-measure} = 2 \ast \frac{Precision\ast Recall}{Precision+Recall}$$

Where *true positive (TP)* defines the total number of positive instances being identified as positive, *true negative (TN)* defines the total number of negative instances being identified as negative, *false positive (FP)* defines the total number of positive instances mistakenly identified as false and *false negative (FN)* defines the total number of positive instances mistakenly identified as negative.

Results
=======

In this section, we will discuss and compare the results of our proposed approach with other baseline models.

### Comparison against the baseline approaches

The top-k accuracy score of the proposed approach and the baseline approaches are presented in \[Table:AccuracyScores\]. Form this table and \[fig:top-k-accuracy\] we make the following observations

-   The average accuracy rate of RNN based model is *45.01%@k=1*, *65.56%@k=5* and *68.55%@k=10*, for the GRU based model is *50.06%@k=1*, *64.38%@k=5* and *73.27%@k=10*, while the proposed approach’s average score is *66.15%@k=1*, *90.68%@k=5* and *93.97%@k=10* which is much higher as compared to the baseline approaches.

-   On average the proposed approach improves the accuracy *(k@1)* by *21.14%* from RNN and *16.09%* from GRU based model.

-   Results suggests that by employing the transfer learning-based *attention* model it significantly improves the model performance.

      K     RNN     GRU    **Proposed**
  -- ---- ------- ------- --------------
      1    40.75   46.86    **62.89**
      5    61.78   66.63    **88.80**
      10   63.69   68.72    **92.22**
      1    53.86   50.19    **62.93**
      5    67.01   71.24    **88.84**
      10   69.25   73.97    **92.16**
      1    46.20   54.81    **66.01**
      5    65.67   72.05    **90.77**
      10   67.63   74.01    **94.29**
      1    41.76   48.08    **68.71**
      5    64.25   68.00    **91.51**
      10   67.06   70.58    **94.47**
      1    42.47   50.39    **70.19**
      5    69.13   73.98    **93.47**
      10   75.14   79.10    **96.73**
      1    45.01   50.06    **66.15**
      5    65.56   70.38    **90.67**
      10   68.55   73.27    **93.97**

  : Top-k Accuracy scores with and without proposed approach[]{data-label="Table:AccuracyScores"}

![Top-k accuracy comparison.[]{data-label="fig:top-k-accuracy"}](img/Top-K-Chart){width="\linewidth"}

Further, to evaluate the performance of the proposed approach we measure the precision, recall and F-measure scores. \[Table:PrecisionScores\] exhibits the precision, recall and F-measure scores. We make the subsequent interpretations from \[Table:PrecisionScores\] and \[fig:f-dist\],

-   The proposed approach’s average F-measure is *68.36*, while RNN and GRU gain much lower score of *39.73* and *46.20* respectively.

-   The proposed approach’s minimum F-measure is much higher than the maximum F-measure of the baseline approaches.

-   The results advise that the anticipated method outclasses the state-of-the-art methods in precision, recall, and F-measure.

                   RNN     GRU    **Proposed**
  -- ----------- ------- ------- --------------
      Precision   30.43   38.91    **63.82**
       Recall     40.75   46.86    **82.89**
      F-measure   34.84   42.52    **72.11**
      Precision   37.29   40.77    **62.39**
       Recall     43.86   50.19    **62.93**
      F-measure   40.31   44.99    **62.65**
      Precision   41.15   46.54    **66.97**
       Recall     46.20   54.81    **66.01**
      F-measure   43.53   50.34    **66.48**
      Precision   37.68   41.23    **69.09**
       Recall     41.76   48.08    **68.71**
      F-measure   39.62   44.39    **68.89**
      Precision   38.11   47.00    **71.02**
       Recall     42.47   50.39    **70.19**
      F-measure   40.17   48.63    **70.60**
      Precision   36.93   42.89    **66.66**
       Recall     43.01   50.07    **70.15**
      F-measure   39.73   46.20    **68.36**

  : Precision, Recall and F-measure comparison with baseline approaches[]{data-label="Table:PrecisionScores"}

![F-measure distribution.[]{data-label="fig:f-dist"}](img/bean){width="0.8\linewidth"}

### RQ2: Comparative analysis for Source Code Suggestion Task

To further quantify the accuracy of the proposed approach for source code suggestion task, we measure the *Mean Reciprocal Rank (MRR)* scores of each model. The MRR is a rank-based evaluation metric which produces a value between *0-1*, where the value closer to *1* indicates a better source code suggestion model. The MRR can be expressed as

$$MRR(C) = \dfrac{1}{|{C}|}\sum_{i=1}^{|{C}|}\dfrac{1}{y^i}$$

where ${C}$ is code sequence and $y^i$ refers to the index of the first relevant prediction. $MRR(C)$ is the average of all sequences $C$ in the test data set.

The results of all models are presented in the \[Table:MRRScores\]. The average MRR score of RNN is *0.5156* and the average score of GRU is *0.5749*, while the average score of the proposed approach is *0.7618* which is much higher. The results suggest that out of four source code suggestions the proposed approach can suggest three predictions at its first rank. From the results, we conclude that the proposed approach significantly outperforms the baseline approaches.

                    RNN      GRU     **Proposed**
  --------------- -------- -------- --------------
   elasticsearch   0.4851   0.5405    **0.7344**
    spring-boot    0.5161   0.5672    **0.7363**
      RxJava       0.5403   0.6085    **0.7619**
    java-design    0.5082   0.5625    **0.7805**
    interviews     0.5284   0.5960    **0.7960**
    **Average**    0.5156   0.5749    **0.7618**

  : MRR scores with and without proposed approach[]{data-label="Table:MRRScores"}

To further validate the statistical significance, we employ the ANOVA One-Way statistical test. We conduct the AVOVA test with its default settings ($\alpha$ = 0.05) using Microsoft Excel, and no modifications were made. Comparing ( \[Table:ANOVA\]) the proposed approach with the best baseline (GRU), we found *F $>$ F-crit* and *P-value $<$ $\alpha$* is true in all cases (Accuracy, MRR, Precision, Recall and F-measure); therefore, we reject the null hypothesis, suggesting that using diverse methods has statistically significant variance in performance.

  ------------------ ---------- --- ------------ ---------- ------------- ----------
  *Accuracy (K@1)*                                                        
  Between Groups        646.416   1      646.416   64.04975   4.35463E-05   5.317655
  Within Groups        80.73924   8    10.092405                          
  Total                727.1552   9                                       
                                                                          
  *MRR*                                                                   
  Between Groups        646.416   1      646.416   64.04975   4.35463E-05   5.317655
  Within Groups        80.73924   8    10.092405                          
  Total                727.1552   9                                       
                                                                          
  *Precision*                                                             
  Between Groups       1412.295   1   1412.29456   108.0003   6.36396E-06   5.317655
  Within Groups        104.6141   8     13.07676                          
  Total                1516.909   9                                       
                                                                          
  *Recall*                                                                
  Between Groups       1008.016   1     1008.016   29.81202   0.000601504   5.317655
  Within Groups        270.4992   8    33.812405                          
  Total                1278.515   9                                       
                                                                          
  *F-measure*                                                             
  Between Groups        1206.92   1   1206.92196    99.9581    8.5015E-06    5.31766
  Within Groups         96.5942   8     12.07428                          
  Total                 1303.52   9                                       
  ------------------ ---------- --- ------------ ---------- ------------- ----------

\[Table:ANOVA\]\
Where, SS = sum of squares, df = degree of freedom, MS = mean square.

### Impact of Normalization

The evaluation outcomes of the projected method for normalized source code and non-normalized source code are presented in \[Table:Normalization\]. From the results, we observe that the normalization of source code improves the model performance significantly. On average the proposed approach with normalization achieves the accuracy score of 66.15@k=1 where without normalization the accuracy drops to 56.27@k=1. From the results ( \[Table:Normalization\]), we conclude that the normalization process significantly affects the model performance.

                    **Accuracy**   **Precision**   **Recall**   **F-measure**   **MRR**
  ---------------- -------------- --------------- ------------ --------------- ---------
     Normalized        66.15           66.66         70.15          68.36       0.7618
   Non-Normalized      56.27           57.25         62.14          54.66       0.6524

  : Impact of Normalization[]{data-label="Table:Normalization"}

Discussion and Future Work
--------------------------

The proposed approach attains the finest performance due to several different reasons. First, the proposed approach takes leverage from pre-trained models by transferring the learned features from them. Second, the *attention* learner fine-tunes the model by paying attention to only task-specific features and does not increase the computational complexity which resulted in better performance. Consequently, transfer learning-based *attention* model has better generalization capability without training the model from scratch.

The broader impact of our work is to show that transfer learning could be beneficial in the domain of source code modeling. This work is the first step in this direction and results encourage future research on it. The work can be improved in several different ways. First, the performance of the proposed approach can be improved by hyper-parameter optimization [@Matuszyk2016]. Second, the proposed approach can be improved by using complex architectures such as transformers [@devlin2018bert] and stacked neural networks [@vincent2010stacked]. Another possible path for improvement is to train the model on an even larger data set. In the future, we consider exploiting these possibilities.

Limitations and Threats to Validity
===================================

A risk to construct validity is the selection of assessment metrics. To alleviate this threat, we use several different evaluation metrics. We use the Top-k accuracy metric as done by former studies [@White2015; @Hindle2012; @Nguyen2018]. We use the precision, recall, and F-measure [@Alon2019] metrics for the evaluation of the proposed approach. These metrics are most generally used for the model evaluation purpose. Moreover, we evaluate the proposed approach with MRR [@Nguyen2018; @Santos2018] metric which is a ranked based metric. Further, to show the statistical significance of the proposed approach we adopt the ANOVA statistical testing.

A risk to internal validity is the employment of the baseline methods. We re-implement the baseline approaches by following the process described in the original manuscripts. To alleviate this risk, we twofold the implementations and results. Conversely, there could be some unobserved inaccuracies. Another risk is the choice of hyper-parameters for deep learning methods. The change in training, validation or testing set or the variation in hyper-parameters may impact the performance of the anticipated method.

A threat to external validity is related to the generality of results. The data set used in this study is collected from *GitHub*, a well-known source code repositories provider. It is not necessary that the projects used in this study represent other languages or Java language source code entirely.

Related Work
============

In this section, we present background study on deep learning, transfer learning and source code language models. We focus on how these approaches can help improve the source code modeling by employing the transfer learning approach.

Source Code Modeling
--------------------

Hindle et al. [@Hindle2012] have shown how natural language processing techniques can help in source code modeling. They provide a *n-gram* based model which helps predict the next code token in *Eclipse IDE*. Tu et al. [@Tu2014], proposed a cache-based language model that consists of an *n-gram* and a *cache*. Hellendoorn et al. [@Hellendoorn2017] further improved the cache-based model by introducing nested locality. Another approach for source code modeling is to use probabilistic context-free grammars(PCFGs) [@Bielik2016]. Allamanis et al. [@Allamanis2014] used a PCFG based model to mine idioms from source code. Maddison et al. [@Maddison2014] used a structured generative model for source code. They evaluated their approach with *n-gram* and *PCFG* based language models and showed how they can help in source code generation tasks. Raychev et al.[@Raychev2016a] applied decision trees for predicting API elements. Chan et al. [@Chan2012] used a graph-based search approach to search and recommend API usages.

Recently there has been an increase in API usage [@Wang2013; @Keivanloo2014; @Dsouza2016b] mining and suggestion. Thung et al. [@Thung2013] introduced a recommendation system for API methods recommendation by using feature requests. Nguyen et al. [@Nguyen2015] proposed a methodology to learn API usages from byte code. Hussain et al. [@Hussain2018] proposed GRU based model for source code suggestion and completion task (completion of a whole line of code). Allamanis et al. [@Allamanis2014] introduced a model which automatically mines source code idioms. A neural probabilistic language model introduced in [@Allamanis2015a] that can suggest names for the methods and classes. Franks et al. [@Franks2015] created a tool for Eclipse named *CACHECA* for source code suggestion using a *n-gram* model. Nguyen et al. [@Nguyen2012] introduced an *Eclipse plugin* which provide code completions by mining the API usage patterns. Chen et al. [@Chen2016] created a web-based tool to find analogical libraries for different languages.

A similar work conducted by Rabinovich et al. [@Rabinovich2017], which introduced an abstract syntax networks modeling framework for tasks like code generation and semantic parsing. Sethi et al. [@Sethi2017a] introduced a model which automatically generate source code from deep Learning-based research papers. [@Allamanis2015], Allamanis et al. proposed a bimodal to help suggest source code snippets with a natural language query. Recently deep learning-based approaches have widely been applied for source code modeling. Such as code summarization [@Iyer2016; @Allamanis2016; @Guerrouj2015], code mining [@Va], clone detection [@Kumar2015], API learning [@Gu2016] etc.

Recently, recurrent neural networks [@zaremba2014; @mikolov2010; @Cho2014] has attracted much attention in various fields such as image generation [@gregor2015], speech recognition [@graves2013speech], text classification [@zhang2015character] and more [@sak2014long; @williams1989learning]. More recently, deep learning has shown great potential for the modeling of source code [@Hussain2018; @Raychev2013a; @White2015; @Gupta2017; @Santos2018]. Raychev et al. [@Raychev2013a] used RNN for the purpose of code completion specifically focusing on suggesting source code method calls. Similarly, White et al. [@White2015] applied RNN based deep neural network for source code completion task. Generally, these approaches are problem-specific which requires training from starch for a different related problem. In this work, we exploit the concept of transfer learning that transfers the learned knowledge from a pre-trained model and then fine-tunes it for a different downstream task.

Transfer Learning {#transfer-learning}
-----------------

Transfer learning as the name suggests intending to transfer knowledge (features) learned in solving one problem into another related problem. Hu et al. [@hu2015] have proposed a transfer metric learning approach for visual recognition in cross-domain datasets. Duan et al. [@Duan2009] have proposed a kernel learning approach for the detection of cross-domain keyframe feature changes. Pan et al. [@pan2008] have proposed a dimensionality reduction method which uses the transfer learning approach by minimizing the distance between distributions between target and source domains. Khan et al. [@khan2019] have proposed a deep transfer learning approach for the detection of breast cancer by using pre-trained GoogLeNet, VGGNet, and ResNet. Huang et al. [@huang2017transfer] have proposed a transfer learning-based approach for Synthetic Aperture Radar (SAR) classification with limited labeled data. Kraus et al. [@KRAUS2017] proposed a decision support system by using deep neural networks and transfer learning for financial disclosures. In this work, we exploit the transfer learning approach for the purpose of source code modeling. Instead of using a single model for transferring knowledge, in this work we use a novel approach that combines two different recurrent neural networks into an attention learner for different source code modeling tasks.

Conclusion
==========

In this work, we proposed a deep learning-based source code language model by using the concept of transfer learning. First, we exploit the concept of transfer learning for neural language-based source code models. Next, we presented RNN and GRU based pre-trained models for the purpose of transfer learning in the domain of source code. Both models are trained on over 13 million code tokens and do not need retraining and can directly be used for the purpose of transfer learning. We evaluated the proposed approach with the downstream task of source code suggestion. We evaluated the proposed approach extensively and compared it with the state-of-the-art models. The extensive evaluation of this work suggests that the proposed approach significantly improves the model’s performance by exploiting the concept of transfer learning.

Acknowledgments {#acknowledgments .unnumbered}
===============

This work was supported by the National Key R&D (grant no. 2018YFB1003902) and Qing Lan Project.

[^1]: <https://docs.oracle.com/javase/specs/jls/se7/html/jls-18.html>

[^2]: Trained Models: <https://github.com/yaxirhuxxain/TransferLearning>
