Googlebot
web crawler used by Google
Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler (to simulate desktop users) and a mobile crawler (to simulate a mobile user).
Behavior
A website will probably be crawled by both Googlebot Desktop and Googlebot Mobile. However, Google transitioned to mobile-first indexing, meaning it predominantly crawls and indexes the web using a smartphone Googlebot. This process began rolling out gradually in 2016 and was completed for all websites by late 2023. The subtype of Googlebot can be identified by looking at the user agent string in the request. However, both crawler types obey the same product token (useent token) in robots.txt, and so a developer cannot selectively target either Googlebot mobile or Googlebot desktop using robots.txt.
Google provides various methods that enable website owners to manage the content displayed in Google's search results. If a webmaster chooses to restrict the information on their site available to a Googlebot, or another spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag <meta name="Googlebot" content="nofollow" /> to the web page. Googlebot requests to web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "googlebot.com".
Begin with the source’s own compact description: “Googlebot” is web crawler used by Google. The dossier treats that line as a proposition to test through Googlebot, crawler and used, not as a finished interpretation.
Why this record matters
The phrase “web crawler used by Google” supplies a clear boundary for inquiry. It also exposes the unanswered questions: who defined that boundary, when it became stable and which sources sit outside it.
Named sources, stable identifiers and responsible institutions provide the strongest route from overview to verifiable evidence. The source revision retrieved here is dated Jun 28, 2026. The linked authority identifier is Q1425771. 1 of 1 selected statements include explicit references; 1 carry qualifiers and 0 use preferred rank. The first chronological checks are 2016 and 2023.
Overview language is designed for orientation and should not be treated as a substitute for the evidence cited beneath it. The source lead contains qualifying language; that uncertainty should survive quotation, summary and reuse. Authority statements aid reconciliation but still require their own references, qualifiers and ranks to be checked.
How to read it
Use the entry as an orientation point, then follow its citations and revision history. Names, dates and institutional relationships should be checked against the original record.
- Subject orientation
- Search vocabulary
- Locating named sources
The closest primary source, responsible institution and strongest cited specialist reference.
Three-step research path
- Establish the record: confirm the title “Googlebot”, its source revision and the description used here.
- Expand the search: follow Googlebot primary sources, Googlebot archive and Googlebot research across catalogues and specialist indexes.
- Test the account: compare the strongest cited source with the responsible institution’s current record and note any disagreement.
Questions for further research
- Which source most directly establishes the central claim about “Googlebot”?
- Which cited source is closest to the event, object or claim?
- What terminology or title could unlock a more precise catalogue search?
Search terms from this dossier
This entry incorporates text from “Googlebot” on English Wikipedia. Contributors are listed in the page history. Text is available under the Creative Commons Attribution-ShareAlike 4.0 License. Selected authority identifiers and statements are retrieved from Wikidata under CC0; their references and qualifiers remain part of the verification path.