  Dark web crawling is a complex process that involves specific methodologies
and techniques to navigate the Tor network and extract data from hidden
services. This study proposes a general dark web crawler designed to extract
pages handling security protocols, such as captchas, efficiently. Our approach
uses a combination of seed URL lists, link analysis, and scanning to discover
new content. We also incorporate methods for user-agent rotation and proxy
usage to maintain anonymity and avoid detection. We evaluate the effectiveness
of our crawler using metrics such as coverage, performance and robustness. Our
results demonstrate that our crawler effectively extracts pages handling
security protocols while maintaining anonymity and avoiding detection. Our
proposed dark web crawler can be used for various applications, including
threat intelligence, cybersecurity, and online investigations.
