  Tabular data is the most widely used data format in machine learning (ML).
While tree-based methods outperform DL-based methods in supervised learning,
recent literature reports that self-supervised learning with Transformer-based
models outperforms tree-based methods. In the existing literature on
self-supervised learning for tabular data, contrastive learning is the
predominant method. In contrastive learning, data augmentation is important to
generate different views. However, data augmentation for tabular data has been
difficult due to the unique structure and high complexity of tabular data. In
addition, three main components are proposed together in existing methods:
model structure, self-supervised learning methods, and data augmentation.
Therefore, previous works have compared the performance without comprehensively
considering these components, and it is not clear how each component affects
the actual performance.
  In this study, we focus on data augmentation to address these issues. We
propose a novel data augmentation method, $\textbf{M}$ask $\textbf{T}$oken
$\textbf{R}$eplacement ($\texttt{MTR}$), which replaces the mask token with a
portion of each tokenized column; $\texttt{MTR}$ takes advantage of the
properties of Transformer, which is becoming the predominant DL-based
architecture for tabular data, to perform data augmentation for each column
embedding. Through experiments with 13 diverse public datasets in both
supervised and self-supervised learning scenarios, we show that $\texttt{MTR}$
achieves competitive performance against existing data augmentation methods and
improves model performance. In addition, we discuss specific scenarios in which
$\texttt{MTR}$ is most effective and identify the scope of its application. The
code is available at https://github.com/somaonishi/MTR/.
