Deep Web

The deep web is the portion of the World Wide Web whose content is not indexed by search engines and so cannot be found by an ordinary web search. It includes pages that sit behind a login, such as internet banking accounts, email inboxes, medical records and company intranets, as well as paywalled articles, private cloud files and results generated on demand from databases. Most of what people do online every day happens on the deep web, and nearly all of it is legal and routine.

The deep web is often confused with the dark web, but the two are different. The dark web is a small subset of the deep web made up of sites deliberately hidden through special software, such as the Tor network, and it is associated with anonymity and, in part, with illegal markets. The rest of the internet, the part search engines can reach, is often called the surface web. The term "deep web" is generally credited to the computer scientist Michael K. Bergman, who used it in a 2001 paper on unindexed online content.

Think of the internet as a large library. The surface web is the open shelves: anyone can walk in and browse. The deep web is everything kept in locked offices and private rooms in the same building, such as staff files, members-only reading rooms and each visitor's personal locker. It is not secret in a sinister way; it is simply not on the public shelves, and a person needs a key or a password to reach it.

A bank statement viewed online is a good example. The page exists on the internet, but no search engine can list it, because only the account holder can log in. The dark web, by contrast, is like a hidden room reached through an unmarked door that needs special directions to find.

• Password-protected accounts. Online banking, brokerage and payment accounts, email, social media inboxes and cloud storage.
• Paid and subscription content. News sites, academic journals and streaming libraries that require a login or payment.
• Dynamic database results. Flight searches, court and land registries, library catalogues and other pages generated only when someone submits a query.
• Internal systems. Company intranets, government portals and the back-office systems that run banks, insurers and payment processors.
• Pages blocked from indexing. Sites that tell search engines not to list them, or that are not linked from anywhere else.

Estimates of the deep web's size vary widely and are hard to verify, but researchers agree it is many times larger than the surface web.

Financial services depend heavily on the deep web. Customer portals, card-management tools, trading platforms and internal ledgers are all kept out of search engines on purpose, both to protect customers and to meet data security and data privacy obligations. Banks protect these areas with authentication, encryption and monitoring. When those defences fail, a data breach can expose deep-web data, and stolen customer details may then be offered for sale on dark web markets, feeding identity theft and account takeovers. Fraud and compliance teams therefore monitor dark web forums for leaked card numbers and credentials linked to their customers.

The dark web part of the deep web has been closely linked with cryptocurrency. Hidden marketplaces reachable through onion routing adopted Bitcoin as a payment method from about 2011, most famously Silk Road, which US authorities shut down in 2013. Because Bitcoin transactions are recorded on a public ledger, investigators were later able to trace many of these payments, and some dark web markets shifted to privacy coins such as Monero. Criminal groups behind ransomware also use dark web sites to publish stolen data and negotiate payments. These links have shaped how regulators view crypto and are one reason exchanges screen deposits for exposure to illicit sources under money-laundering rules.

Popular coverage often uses "deep web" to mean something dangerous or illegal, which overstates the risk: most people use the deep web many times a day simply by checking email or a bank balance. Accessing deep-web content is legal wherever the user is authorized, while accessing other people's accounts without permission is a crime in most countries. Using privacy tools such as Tor is itself legal in many countries and is relied on by journalists, activists and people living under censorship, though some governments restrict it. The boundary between surface and deep web also shifts as search engines change what they index and as sites change their settings. Artificial-intelligence crawlers and paywall agreements are also changing which content is reachable and by whom. For individuals, the practical lessons are to protect passwords, use strong authentication and treat unexpected requests for login details as possible phishing.