The Deep Web: The Internet's Dark Side
The Web's Hidden Content.
Tor: The Onion Router.
The Dark Side.
The Bright Side.
Pierluigi Paganini
Originally published in the Britannica Book of the Year. Presented as archival content. Learn more.
This article was originally published in the Britannica Book of the Year, an annual print publication that provides an overview of the year's most-notable people and events. Unlike most articles on Britannica.com, Book of the Year articles are not reviewed and revised after their initial publication. Rather, they are presented on the site as archival content, intended for historical reference only.
In mid-2014 it was estimated that more than three billion people (about 42% of the world's population) used the Internet. The majority of those people accessed the World Wide Web by using the same Web browsers: Microsoft Internet Explorer, Mozilla Firefox, Apple Safari, and Google Chrome. Few Internet users, however, are familiar with the Deep Web—also known as the Hidden Web, Deepnet, or Invisible Web—that portion of the Web content that is not indexed by the widely used standard search engines. The Deep Web differs from the dark Internet (sites that have been dropped or otherwise concealed and cannot be reached through conventional search engines) and from Darknets (private networks for sharing files between trusted peers that are accessed through nonstandard protocols). Modern search engines index only a small portion of the content on the World Wide Web, which satisfies the needs of the majority of users, who remain unaware of the vast amount of material to be found in the Deep Web.
Silk Road 2.0 seized noticeThis notice was posted on the Deep Web site Silk Road 2.0 in November 2014 after the hidden black-market site was seized by authorities in the U.S. and Europe. Many conventional Internet users were first alerted to the existence of the Deep Web (and its darker side) in 2013 when the original Silk Road was similarly shut down. David Colbran/Alamy
Silk Road 2.0 sale of speedThis screen shot shows illegal drugs for sale—in this case 28 g (1 oz) of speed available to be shipped from Germany—on the black-market Deep Web site Silk Road 2.0, which was shut down by U.S. and European law-enforcement agencies in November 2014. Boris Roessler—picture-alliance/dpa/AP Images
It is impossible to estimate the real size of the Deep Web, because the majority of data is not accessible to the casual user, and new content is always being added. Michael Bergman, the founder of the intelligence company BrightPlanet, compared searching on the Internet to dragging a net across the surface of the ocean: search engines are able to analyze the surface content but fail to recognize that there is a wealth of information hidden below. In a 2001 study, Bergman hypothesized that the overall content in the Deep Web consisted of about 7.5 petabytes. The Deep Web thus could have been 4,000–5,000 times larger than the so-called Surface Web at that time, and it was estimated to be growing at an exponential rate.
Search engines use automated programs, or bots, called web crawlers, to browse the World Wide Web automatically and index its content for easy access by users. This process is not effective, however, for hidden resources that fall into certain discrete categories:
Dynamic content: dynamic pages that are returned in response to a specific query or accessed only through an online form, particularly if text fields or other open-domain input elements are used
Unlinked content: pages that are not linked to other Web sites, which may prevent the Web crawlers from identifying the content
Private Web: Web sites that are protected by authentication mechanisms
Contextual Web: pages with content that varies for different access contexts (for example, ranges of client IP addresses)
Limited access content: sites that limit access to their pages in a technical way (for example, those that require a user to respond to CAPTCHA images or code words in visually distorted type)
Scripted content: pages that are reached only through links produced by the computer language JavaScript and content that is downloaded by using specialized software, such as the animation software Adobe Flash
Non-HTML/text content: encoded videos or other image files and formats that search engines do not handle
Non-HTTP content: sites that do not use the standard Hypertext Transfer Protocol (HTTP), such as those that use the earlier Gopher protocol that is not accessed through modern search engines
Users can preserve their anonymity online and make their content inaccessible through the Internet, using networks that provide anonymity. The most common of these are Tor (originally The Onion Router) and the Invisible Internet Project (I2P).
Tor is the most-popular system designed to provide anonymity for online users. Tor users employ a client software to route Internet traffic through a worldwide volunteer network of servers. The complex network of Tor relays and the multiple layers of data encryption conceal personal information, including the IP address of the user's computer, and make it much more difficult to trace that user's location or track online activities.
The Tor Project was originally sponsored by the U.S. Naval Research Laboratory and was intended to enable encrypted communications in the military and intelligence communities. During 2004–05 it was financially supported by the Electronic Frontier Foundation. By 2014 the Tor Project was being sponsored by such entities as the National Science Foundation, the Bureau of Democracy, Human Rights, and Labor of the U.S. Department of State, the Ford Foundation, and Google, Inc.'s open source programs office.
When a Tor user wishes to access a resource on the Web, he makes an encrypted connection to a centralized directory server that contains the addresses of Tor nodes. The Tor client then uses the address list provided to connect to a random node (the entry node), through an encrypted connection. The entry node then makes another encrypted connection to a randomly selected second node, which in turn connects to a random third Tor node. The process continues until the client reaches a node (the exit node) that is connected to the destination sought. Each single node along the way ignores the path of the data. The Tor software on the user's computer negotiates a separate set of encryption keys for each hop in order to implement multiple levels of encryption and avoid traffic eavesdropping. To ensure anonymity every connection has a fixed duration to avoid the threat of actors that can use a statistical analysis to track users. For each connection the client software changes the entry node, and no node is used twice in one connection.
Tor software can be used to access any online Web site. It is also possible to use a live operating system, such as the free software Tails (the amnesiac incognito live system), to navigate the Deep Web anonymously and not leave any trace on the host computer. Tor users, for example, can employ an entry point such as the Hidden Wiki, which includes a list of accessible links, grouped by category. Domain names in the Tor network use the suffix .onion instead of the classic Web site extensions, which include .com, .org, and .gov.