  The extraction of cyber threat intelligence (CTI) from open sources is a
rapidly expanding defensive strategy that enhances the resilience of both
Information Technology (IT) and Operational Technology (OT) environments
against large-scale cyber-attacks. While previous research has focused on
improving individual components of the extraction process, the community lacks
open-source platforms for deploying streaming CTI data pipelines in the wild.
To address this gap, the study describes the implementation of an efficient and
well-performing platform capable of processing compute-intensive data pipelines
based on the cloud computing paradigm for real-time detection, collecting, and
sharing CTI from different online sources. We developed a prototype platform
(TSTEM), a containerized microservice architecture that uses Tweepy, Scrapy,
Terraform, ELK, Kafka, and MLOps to autonomously search, extract, and index
IOCs in the wild. Moreover, the provisioning, monitoring, and management of the
TSTEM platform are achieved through infrastructure as a code (IaC). Custom
focus crawlers collect web content, which is then processed by a first-level
classifier to identify potential indicators of compromise (IOCs). If deemed
relevant, the content advances to a second level of extraction for further
examination. Throughout this process, state-of-the-art NLP models are utilized
for classification and entity extraction, enhancing the overall IOC extraction
methodology. Our experimental results indicate that these models exhibit high
accuracy (exceeding 98%) in the classification and extraction tasks, achieving
this performance within a time frame of less than a minute. The effectiveness
of our system can be attributed to a finely-tuned IOC extraction method that
operates at multiple stages, ensuring precise identification of relevant
information with low false positives.
