Robots.txt
standard used to advise web crawlers and scrapers not to index a web page or site
The Robots Exclusion Protocol (often referred to by the filename used to implement it, robots.txt) is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with security through obscurity. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate server overload. In the 2020s, websites began denying bots that collect information for generative AI.
The "robots.txt" file can be used in conjunction with sitemaps, another robot inclusion standard for websites.
History
The standard was proposed by Martijn Koster, when working for Nexor in February 1994 on the www-talk mailing list, the main communication channel for WWW-related activities at the time. Charles Stross claims to have provoked Koster to suggest robots.txt, after he wrote a badly behaved web crawler that inadvertently caused a denial-of-service attack on Koster's server.
The standard, initially RobotsNotWanted.txt, allowed web developers to specify which bots should not access their website or which pages bots should not access. The internet was small enough in 1994 to maintain a complete list of all bots; server overload was a primary concern.
Begin with the source’s own compact description: “Robots.txt” is standard used to advise web crawlers and scrapers not to index a web page or site. The dossier treats that line as a proposition to test through Robots, standard and used, not as a finished interpretation.
Why this record matters
The phrase “standard used to advise web crawlers and scrapers not to index a web page or site” supplies a clear boundary for inquiry. It also exposes the unanswered questions: who defined that boundary, when it became stable and which sources sit outside it.
The citation trail is more important than the brevity of the summary: it shows where individual claims can be examined in context. The source revision retrieved here is dated Sep 21, 2026. The linked authority identifier is Q80776. 1 of 1 selected statements include explicit references; 1 carry qualifiers and 0 use preferred rank. The first chronological checks are 1994.
A concise general-reference account can conceal disagreements about scope, terminology or the weight assigned to individual sources. The lead is largely declarative, so disagreement and counter-evidence require a deliberate search beyond the opening account. Authority statements aid reconciliation but still require their own references, qualifiers and ranks to be checked.
How to read it
Use the entry as an orientation point, then follow its citations and revision history. Names, dates and institutional relationships should be checked against the original record.
- Subject orientation
- Search vocabulary
- Locating named sources
The closest primary source, responsible institution and strongest cited specialist reference.
Three-step research path
- Establish the record: confirm the title “Robots.txt”, its source revision and the description used here.
- Expand the search: follow Robots.txt primary sources, Robots.txt archive and Robots research across catalogues and specialist indexes.
- Test the account: compare the strongest cited source with the responsible institution’s current record and note any disagreement.
Questions for further research
- Which source most directly establishes the central claim about “Robots.txt”?
- Which institution is responsible for the underlying evidence?
- Which cited source is closest to the event, object or claim?
Search terms from this dossier
This entry incorporates text from “Robots.txt” on English Wikipedia. Contributors are listed in the page history. Text is available under the Creative Commons Attribution-ShareAlike 4.0 License. Selected authority identifiers and statements are retrieved from Wikidata under CC0; their references and qualifiers remain part of the verification path.