  This paper introduces the 2nd place solution for the Riiid! Answer
Correctness Prediction in Kaggle, the world's largest data science competition
website. This competition was held from October 16, 2020, to January 7, 2021,
with 3395 teams and 4387 competitors. The main insights and contributions of
this paper are as follows. (i) We pointed out existing Transformer-based models
are suffering from a problem that the information which their query/key/value
can contain is limited. To solve this problem, we proposed a method that uses
LSTM to obtain query/key/value and verified its effectiveness. (ii) We pointed
out 'inter-container' leakage problem, which happens in datasets where
questions are sometimes served together. To solve this problem, we showed
special indexing/masking techniques that are useful when using RNN-variants and
Transformer. (iii) We found additional hand-crafted features are effective to
overcome the limits of Transformer, which can never consider the samples older
than the sequence length.
