  Collecting data on underground criminal communities is highly valuable both
for security research and security operations. Unfortunately these communities
live within a constellation of diverse online forums that are difficult to
infiltrate, may adopt crawling monitoring countermeasures, and require the
development of ad-hoc scrapers for each different community, making the
endeavour increasingly technically challenging, and potentially expensive. To
address this problem we propose THREAT/crawl, a method and prototype tool for a
highly reusable crawler that can learn a wide range of (arbitrary) forum
structures, can remain under-the-radar during the crawling activity and can be
extended and configured at the user will. We showcase THREAT/crawl capabilities
and provide prime evaluation of our prototype against a range of active, live,
underground communities.
