  The dark web has become notorious for its association with illicit activities
and there is a growing need for systems to automate the monitoring of this
space. This paper proposes an end-to-end scalable architecture for the early
identification of new Tor sites and the daily analysis of their content. The
solution is built using an Open Source Big Data stack for data serving with
Kubernetes, Kafka, Kubeflow, and MinIO, continuously discovering onion
addresses in different sources (threat intelligence, code repositories, web-Tor
gateways, and Tor repositories), downloading the HTML from Tor and
deduplicating the content using MinHash LSH, and categorizing with the BERTopic
modeling (SBERT embedding, UMAP dimensionality reduction, HDBSCAN document
clustering and c-TF-IDF topic keywords). In 93 days, the system identified
80,049 onion services and characterized 90% of them, addressing the challenge
of Tor volatility. A disproportionate amount of repeated content is found, with
only 6.1% unique sites. From the HTML files of the dark sites, 31 different
low-topics are extracted, manually labeled, and grouped into 11 high-level
topics. The five most popular included sexual and violent content,
repositories, search engines, carding, cryptocurrencies, and marketplaces.
During the experiments, we identified 14 sites with 13,946 clones that shared a
suspiciously similar mirroring rate per day, suggesting an extensive common
phishing network. Among the related works, this study is the most
representative characterization of onion services based on topics to date.
