Should hotels block AI crawlers?
Short answer
Most hotels should not block the crawlers that AI search products use to show websites in their answers, because blocking them can keep your site out of those results. Training crawlers are a separate decision: you can allow search while opting out of model training. For example, OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training as independent settings, and Google-Extended controls some uses of content for Google's AI models without affecting Google Search. Decide per crawler in robots.txt, and check your firewall isn't blocking them by accident.
By Lodging Engine Editorial Team· Reviewed by Eric Zwanenburg· Last reviewed
Know which crawler does what
| Crawler token | Operator | Purpose (per the operator's documentation) |
|---|---|---|
Googlebot | Crawling for Google Search, including AI Overviews and AI Mode | |
Google-Extended | Controls use of content for training and grounding some of Google's AI models; not a Search ranking signal | |
OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search results |
GPTBot | OpenAI | Content that may be used for training foundation models |
PerplexityBot | Perplexity | Surfacing and linking sites in Perplexity search; not used for training foundation models |
Operators update these lists; check their documentation before changing your rules.
What blocking changes
- Blocking `Googlebot` removes you from Google Search — never do this on a live hotel site.
- Blocking `OAI-SearchBot`: OpenAI says opted-out sites won't be shown in ChatGPT search answers (they may still appear as navigational links).
- Blocking `GPTBot` or `Google-Extended` is a training-data choice; per the operators, it doesn't remove you from their search results.
- User-initiated fetchers (such as ChatGPT-User or Perplexity-User) act on a person's request and may not follow robots.txt in the same way — check each operator's documentation.
Example robots.txt (allow search, opt out of training)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Example only. Robots.txt controls crawling, not indexing — Google notes it isn't a way to keep a page out of search results.
Check your firewall and CDN
Bot protection settings at hosting or CDN level can block AI search crawlers even when robots.txt allows them. If you want to appear in AI search, review those settings too.
How to decide
- Want to appear in AI search answers? Allow the search crawlers.
- Concerned about training use of your content? Disallow the training tokens.
- Unsure? Allowing search while blocking training is a common middle ground.
Common mistakes
- A blanket
Disallow: /for unknown bots that catches search crawlers. - Copying another site's robots.txt without understanding each rule.
- Forgetting staging rules (
Disallow: /) on the live site.
Checklist
- robots.txt reviewed rule by rule
- Search crawlers allowed (Googlebot, OAI-SearchBot, PerplexityBot)
- Training opt-outs set deliberately
- CDN / firewall bot settings checked
- Changes re-checked after site launches
Sources
- Overview of OpenAI crawlers — OpenAI
- Perplexity crawlers — Perplexity
- List of Google's common crawlers — Google Search Central
- Robots.txt introduction and guide — Google Search Central
- AI features and your website — Google Search Central