  As various forms of fraud proliferate on Ethereum, it is imperative to
safeguard against these malicious activities to protect susceptible users from
being victimized. While current studies solely rely on graph-based fraud
detection approaches, it is argued that they may not be well-suited for dealing
with highly repetitive, skew-distributed and heterogeneous Ethereum
transactions. To address these challenges, we propose BERT4ETH, a universal
pre-trained Transformer encoder that serves as an account representation
extractor for detecting various fraud behaviors on Ethereum. BERT4ETH features
the superior modeling capability of Transformer to capture the dynamic
sequential patterns inherent in Ethereum transactions, and addresses the
challenges of pre-training a BERT model for Ethereum with three practical and
effective strategies, namely repetitiveness reduction, skew alleviation and
heterogeneity modeling. Our empirical evaluation demonstrates that BERT4ETH
outperforms state-of-the-art methods with significant enhancements in terms of
the phishing account detection and de-anonymization tasks. The code for
BERT4ETH is available at: https://github.com/git-disl/BERT4ETH.
