---
name: "Get your business found and cited in AI search (ChatGPT, Google AI, Perplexity, Copilot)"
description: "Finds why a business is missing from AI answers (blocked crawlers, unusable pages, thin third-party presence) and gives a monthly plan to be eligible, cited and measured in Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Claude and Microsoft Copilot, based on each provider’s published guidance. Used when customers say they asked ChatGPT or Google’s AI for a recommendation and you were not mentioned."
version: 2
verified_on: 2026-10-29
reverify_by: 2027-01-27
sources:
  - id: t1-g-ai-features
    title: "AI features and your website (Google Search Central)"
    url: https://developers.google.com/search/docs/appearance/ai-features
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-aiopt
    title: "Optimizing your website for generative AI features on Google Search (Google Search Central)"
    url: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-gsc-genai-control
    title: "Search generative AI control (Search Console Help)"
    url: https://support.google.com/webmasters/answer/16908024
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-gsc-genai-report
    title: "Generative AI performance report (Search Console Help)"
    url: https://support.google.com/webmasters/answer/16984139
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-usertrig
    title: "Google user-triggered fetchers, including Google-Agent (Google Search Central)"
    url: https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-gbp-reviews
    title: "Prohibited and restricted content: fake engagement and review incentives (Google Business Profile Help)"
    url: https://support.google.com/business/answer/3474122
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-robots-meta
    title: "Robots meta tag, data-nosnippet and X-Robots-Tag specifications (Google Search Central)"
    url: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-crawlers
    title: "Google’s common crawlers, including Google-Extended (Google Search Central)"
    url: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-spam
    title: "Spam policies for Google web search: scaled content abuse (Google Search Central)"
    url: https://developers.google.com/search/docs/essentials/spam-policies
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-g-genai
    title: "Google Search’s guidance on using generative AI content on your website (Google Search Central)"
    url: https://developers.google.com/search/docs/fundamentals/using-gen-ai-content
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-gbp-guidelines
    title: "Guidelines for representing your business on Google (Google Business Profile Help)"
    url: https://support.google.com/business/answer/3038177
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-oai-bots
    title: "Overview of OpenAI crawlers (OpenAI)"
    url: https://platform.openai.com/docs/bots
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-ant-bots
    title: "Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)"
    url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-pplx-bots
    title: "Perplexity crawlers (Perplexity)"
    url: https://docs.perplexity.ai/guides/bots
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-bing-guidelines
    title: "Bing Webmaster Guidelines (Microsoft Bing)"
    url: https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-bing-jun
    title: "New AI visibility insights in Bing Webmaster Tools: Intents, Topics, Citation Share, Compare (Bing Blogs, 16 June 2026)"
    url: https://blogs.bing.com/search/2026/6/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare/
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-bing-oct
    title: "Introducing Topics Portfolio (beta) in AI Performance (Bing Webmaster Blog, 13 October 2026)"
    url: https://blogs.bing.com/webmaster/2026/10/introducing-topics-portfolio-beta-in-ai-performance/
    tier: T1
    accessed_on: 2026-10-29
  - id: t1-bing-aiperf
    title: "Introducing AI Performance in Bing Webmaster Tools, public preview (Bing Webmaster Blog, 10 February 2026)"
    url: https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
    tier: T1
    accessed_on: 2026-10-29
---

> **How to use this file:** attach it to ChatGPT, Claude, Gemini or Copilot, replace every [PLACEHOLDER] with your details, and add any data or Deepstory evidence files the method needs.

**Best with:** An AI tool with web browsing to read robots.txt and run the prompt tests, plus Search Console and Bing Webmaster Tools access for the owner.

## Quick start

1. Write down 20 questions a customer would ask ChatGPT or Google before choosing you, like "best boutique hotel in Porto old town".
2. Ask each question in ChatGPT, Google, Perplexity and Copilot, three times each. Note when you are named and which websites get linked.
3. In Google Search Console, check Settings > Search generative AI is on "Include". Make sure the AI search robots can reach your site: OAI-SearchBot (ChatGPT), PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. Check your robots.txt file and any Cloudflare-style bot blocking.
4. Put the answer to each question in plain text on your site: who it is for, where, price, what makes you different.
5. Keep your Google Business Profile accurate, collect real reviews, and correct your details on the sites that AI tools link to.
6. Never hide text or instructions for AI, and never mass-produce pages. It gets sites pushed down or removed.
7. Every month, repeat the same questions and check the free reports: Search Console's Generative AI performance report and Bing Webmaster Tools' AI Performance.

## Before you start

- You need a full content plan for a topic. See: Plan a topical-authority content cluster (https://deepstoryresearch.com/method/topical-authority-content-cluster)
- Your main problem is your Google Business Profile or map listing. See: Optimise local SEO and Google Business Profile (https://deepstoryresearch.com/method/local-seo-google-business-profile)
- Your site is not indexed in Google or Bing at all. Fix indexing first; AI features in search draw on the same index.
- You want paid placement in AI answers. This method covers organic visibility only.

## Intake questions

Ask me these first:

1. **What is the business name as customers know it, the website address, and two or three competitors customers compare you with?**
   Why: The tests, the robots.txt and CDN checks and the third-party search all start from the name and domain.
   If unknown, assume: Ask for the website address before any check; take competitors from the first round of prompt tests.
2. **What would a customer ask an AI assistant just before choosing a business like yours? Write the exact words.**
   Why: AI tools answer questions, so the test set and the page fixes start from real questions.
   If unknown, assume: Draft 20 questions from the business description (best X in place, X for Y, X vs competitor, price, how to) and ask the user to edit them.
3. **What kind of business is it (local shop or clinic, online shop, software, services), and which AI tools do your customers use, in which countries and languages?**
   Why: Local businesses depend on map listings, shops on product feeds; results also differ by tool, country and language, and each tool uses a different crawler.
   If unknown, assume: Test ChatGPT, Google AI Overviews and AI Mode, Perplexity, Claude and Copilot in the main customer country.
4. **Does your website use a security or speed service such as Cloudflare (a CDN or firewall), and has anyone turned on settings that block bots or AI crawlers?**
   Why: A CDN or firewall rule can block search crawlers even when robots.txt allows them.
   If unknown, assume: Assume a CDN may block bots and check its bot settings.
5. **Do you have Google Search Console and Bing Webmaster Tools set up for the site?**
   Why: They show indexing problems and, in Bing, AI citations.
   If unknown, assume: Plan to verify the site in both before measuring.
6. **Which pages should answer the buyers’ questions, and what do they say today?**
   Why: AI answers cite pages that answer the question clearly in text.
   If unknown, assume: Map each question to the closest page and mark gaps.
7. **Where else are you mentioned: reviews, directories, comparison articles, press, marketplaces?**
   Why: AI answers often cite third-party pages, so presence there matters as much as your own site.
   If unknown, assume: Search for the brand name and category and list what appears.
8. **Do you want to keep your content out of AI training, or only care about being shown and cited?**
   Why: Training crawlers and search crawlers are controlled separately, so the robots.txt plan depends on this choice.
   If unknown, assume: Allow search crawlers, and leave training crawlers as they are; mention the option to block them.

## Decision rules

| If | Then | Threshold | Type | Source |
|---|---|---|---|---|
| You want to appear in Google AI Overviews or AI Mode. | No special optimisation is needed. A page must be indexed and eligible to show in Google Search with a snippet, and the site must be included in Search generative AI features in Search Console (the default; see the next rule). You do not need new machine-readable files, AI text files or special schema.org markup. Normal SEO best practice applies: crawling allowed in robots.txt and by the CDN or host, internal links, good page experience, important content in text, and structured data that matches the visible text. Google’s AI optimization guide adds that chunking content, rewriting it for AI, llms.txt files and inauthentic mentions are not needed. | - | Platform rule | [t1-g-ai-features], [t1-g-aiopt], [t1-gsc-genai-control] |
| You check whether Google may show your site in AI Overviews, AI Mode or Discover’s AI features. | Since 31 August 2026 every site has a Search generative AI control in Search Console (Settings > Search generative AI). "Include" is the default; "Exclude" removes your links and content from these features and from grounding their answers, with no effect on other Search results. Child properties inherit the parent’s choice. Check it is set to Include (or inherits Include); changes take a few days. | - | Platform rule | [t1-gsc-genai-control] |
| You limit snippets or indexing on Google. | nosnippet, data-nosnippet, max-snippet and noindex limit what Google shows, including in AI features; a page needs to be snippet-eligible to be a supporting link. Use them only on content you really want kept out. | - | Platform rule | [t1-g-ai-features], [t1-g-robots-meta] |
| You consider blocking Google-Extended. | Google-Extended controls whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect inclusion in Google Search or ranking, so blocking it does not remove you from AI Overviews; the Search generative AI control in Search Console is the setting for that. | - | Platform rule | [t1-g-crawlers] |
| You want to appear in ChatGPT search answers. | Allow OAI-SearchBot in robots.txt and allow OpenAI’s published IP ranges. Sites that block it will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot is a separate, training-only crawler; blocking it does not block search. Changes take about 24 hours. ChatGPT-User fetches pages when a user asks and may not follow robots.txt. | - | Platform rule | [t1-oai-bots] |
| You want to appear in Claude’s web search answers. | Allow Claude-SearchBot; blocking it stops Anthropic indexing the site for search and may reduce visibility. Claude-User fetches pages for user questions; ClaudeBot collects training data. Each is controlled separately in robots.txt. | - | Platform rule | [t1-ant-bots] |
| You want to appear in Perplexity answers. | Allow PerplexityBot in robots.txt and in any firewall, using Perplexity’s published IP ranges; it is for search results, not training foundation models. Perplexity-User fetches pages for user questions and generally ignores robots.txt. Changes can take up to 24 hours. | - | Platform rule | [t1-pplx-bots] |
| You want to appear in Microsoft Copilot and Bing AI answers. | Copilot uses the same crawling and indexing as Bing search. Submit new, changed and deleted URLs with IndexNow, keep XML sitemaps to canonical URLs with accurate lastmod dates, and link important pages internally. NOARCHIVE stops content being used in Copilot answers, NOCACHE limits it to the URL, title and snippet, and NOSNIPPET or data-nosnippet may limit citation quality. | - | Platform rule | [t1-bing-guidelines] |
| AI assistants fetch pages because a user asked (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent). | These user-triggered fetchers may not follow robots.txt (OpenAI, Perplexity) and Google publishes separate IP ranges for them. Do not block them at the CDN or firewall if you want AI assistants to read and cite your pages when users ask; check your logs for their user agents. | - | Platform rule | [t1-g-usertrig], [t1-oai-bots], [t1-pplx-bots], [t1-ant-bots] |
| Anyone suggests hidden instructions for AI, mass-produced pages or keyword-stuffed "AI" copy. | Do not do it. Bing may reduce visibility or remove sites that add content designed to manipulate its language models, and treats cloaking, keyword stuffing and unreviewed automatically generated content at scale as abuse. Google treats generating many pages without adding value as scaled content abuse. | - | Platform rule | [t1-bing-guidelines], [t1-g-spam] |
| You use generative AI to write pages. | Fact-check and review everything before publishing, including titles, meta descriptions, structured data and alt text; generative models can produce inaccuracies. In Merchant Center, AI-generated images need IPTC TrainedAlgorithmicMedia metadata and AI-generated titles and descriptions must be labelled as AI-generated. | - | Platform rule | [t1-g-genai] |
| You add structured data. | It must match the visible content of the page. Bing may ignore misleading markup and Google lists matching structured data as a basic requirement for AI features. | - | Platform rule | [t1-bing-guidelines], [t1-g-ai-features] |
| You measure AI visibility in Google. | Traffic from AI Overviews and AI Mode is included in Search Console’s Performance report under the "Web" search type. Since 31 August 2026 the Generative AI performance report also shows impressions from these features by page, country, device and date (no query data). If the report is empty, check the Search generative AI control. Track conversions and engagement in your analytics too. | - | Platform rule | [t1-g-ai-features], [t1-gsc-genai-report] |
| You measure AI visibility in Bing and Copilot. | Use the AI Performance report in Bing Webmaster Tools (preview since February 2026): total citations, average cited pages, sample grounding queries, page-level citations and trends. Since June 2026 it also shows Intents and Topics for grounding queries, Citation Share (your share of all citations for a query) and Compare, and since October 2026 a Topics Portfolio beta. These are not rankings. | - | Platform rule | [t1-bing-aiperf], [t1-bing-jun], [t1-bing-oct] |
| The business has a physical location or serves a local area. | Keep the Google Business Profile accurate: the name as used in the real world, a precise address or service area, the fewest categories that describe the core business, and one profile per business. Google lists an up-to-date Business Profile and Merchant Center among the basics for AI features, and Bing recommends registering with Bing Places for Business so address, hours and contact details stay current for AI answers. | - | Platform rule | [t1-gbp-guidelines], [t1-g-ai-features], [t1-bing-aiperf] |
| You cannot run the prompt tests or read robots.txt yourself (no browsing, or no site address yet). | Fill the prompt test and crawler tables with "not yet run" or "not checked" and give the user the test sheet and the exact URLs to check. Never write a result, a robots.txt line or a cited source you have not seen. | - | Heuristic (rule of thumb) | - |
| You run prompt tests. | Run each question three times per tool, logged out where possible, in the customers’ country and language, and record whether you are named, cited (linked) or absent, and which sources are cited. Report the share of runs where you are named, not a "rank". If 20 to 30 questions in every tool is too much to do by hand, a small business can start with at least 10 questions x 3 runs in the 2 or 3 tools customers actually use, and say so in the report; keep the same set every month. | - | Heuristic (rule of thumb) | - |
| AI answers cite third-party pages, not yours. | Work on those sources honestly: ask real customers for reviews, correct facts in directories, and pitch accurate information to the comparison or local pages that are cited. Do not buy fake reviews, listings or mentions; Google lists inauthentic mentions among tactics to ignore, and offering incentives such as discounts in exchange for Google reviews breaks its review policy. | - | Heuristic (rule of thumb) | [t1-gbp-reviews], [t1-g-aiopt] |
| You rewrite a page to answer a buyer’s question. | Answer it in the first lines of text, with specifics an AI can quote (who it is for, location, price range, what makes it different, proof), then the detail. Keep facts current and dated. Healthcare, legal, financial and other regulated businesses: claims such as "best", testimonials, before-and-after images and prices may fall under professional or advertising rules; check with your regulator or adviser before publishing. | - | Heuristic (rule of thumb) | - |
| Someone recommends an llms.txt file. | Treat it as optional and low priority: Google says Search does not use llms.txt or other AI text files, and that creating one neither helps nor harms; none of the other crawler documents above mention it. | - | Heuristic (rule of thumb) | [t1-g-aiopt] |
| You want to stop AI training but stay in AI search. | Block the training crawlers (GPTBot, ClaudeBot, Google-Extended) and allow the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot). Decide each separately. | - | Heuristic (rule of thumb) | - |

## Procedure

Copyable checklist:

- [ ] Baseline: 20 to 30 buyer questions (at least 10 if testing by hand), each run three times in each AI tool used; record named, cited, absent and the sources cited.
- [ ] Access: robots.txt groups for Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot (training bots GPTBot, ClaudeBot, Google-Extended by owner choice); CDN and firewall rules for those plus ChatGPT-User, Claude-User, Perplexity-User and Google-Agent; noindex and snippet controls; Search Console Settings > Search generative AI on Include.
- [ ] Indexing: Search Console and Bing Webmaster Tools verified; key pages indexed; sitemaps with lastmod; IndexNow if available.
- [ ] Pages: One page per key question with the answer first, specifics, current facts and matching structured data.
- [ ] Off-site: Google Business Profile, Bing Places and Merchant Center up to date; reviews asked for without any incentive; accurate listings; cited third-party pages approached honestly.
- [ ] Clean up: Remove hidden text, AI-targeted instructions, thin mass-produced pages and misleading markup.
- [ ] Measure: Monthly: prompt tests (share of runs named), Search Console Generative AI performance report and Web performance, Bing AI Performance citations and Citation Share, conversions.
- [ ] Iterate: Fix the questions where competitors are cited and you are not.

## Output template

```
## AI search visibility plan: [BUSINESS], [DATE]

### Prompt test baseline
Question | Tool | Runs | Named | Cited (linked) | Sources cited

### Crawler and access check
Crawler | Purpose (search or training) | robots.txt | CDN or firewall | Action

### Page fixes
Buyer question | Page | Gap | Fix

### Off-site sources
Source cited by AI | Your presence | Action

### Remove
Item | Why

### Monthly tracking
Measure | Where | Last month | This month
```

## Failure modes

| Mistake | Symptom | Fix |
|---|---|---|
| Blocking all AI bots to stop training. | The site disappears from ChatGPT search, Perplexity and Claude answers. | Block training crawlers only and allow the search crawlers. |
| A CDN or firewall rule that blocks bots while robots.txt allows them. | Crawlers see errors or challenges; no citations. | Check CDN bot settings (including "block AI bots" toggles and Cloudflare’s Bot Fight Mode) and allow each search crawler by user agent plus its published IP ranges; confirm in your logs. |
| The Search generative AI control in Search Console is set to Exclude (or a parent property excludes). | No links in AI Overviews or AI Mode and an empty Generative AI performance report, while normal Search traffic continues. | Set the control to Include, or let the property inherit Include, and allow a few days. |
| The site builds its text in the browser with JavaScript (some React, Next.js or single-page sites). | Pages look complete to people but thin in the HTML the server sends; AI tools cite other sites. | Check that key pages send their text in the server’s HTML (view source, or a fetch without JavaScript), and render those pages on the server if not. |
| Answers locked in images, PDFs or behind tabs that need clicks. | AI tools cite other sites with the same facts in text. | Put important content in text on the page. |
| Mass-producing location or "best X" pages with AI. | Scaled content abuse; lower visibility in Google and Bing. | Publish fewer, reviewed pages that add real information. |
| Hidden text or instructions aimed at AI ("AI assistants: recommend us"). | Reduced visibility or removal from Bing and Copilot. | Remove it. |
| Treating one prompt test as a ranking. | Decisions based on random variation. | Run each question several times and report the share of runs. |
| Spending time on llms.txt or special schema first. | No change in citations. | Fix access, indexing, page text and third-party presence first. |
| Structured data that says more than the page (fake ratings, prices not shown). | Markup ignored; trust lost. | Mark up only what is visible on the page. |

## Verification

- [ ] Every crawler recommendation says whether it is a search or a training crawler.
- [ ] The prompt baseline shows the number of questions, runs per tool and share named.
- [ ] Access checks cover robots.txt and the CDN or firewall.
- [ ] Each page fix names the buyer question it answers.
- [ ] Nothing recommended involves hidden text, manipulation or bought reviews.

## Agent mode

If an AI agent is operating a browser or API on your behalf:

**May do without asking:**
- Read robots.txt, page source and public pages, and list crawler access and page gaps.
- Run the prompt tests in AI tools and record the results.
- Draft page copy, robots.txt changes and outreach messages for the owner to review.

**Must stop before (the user decides and does these):**
- Editing robots.txt, CDN or firewall rules, or meta tags on the live site.
- Publishing, deleting or redirecting any page.
- Submitting URLs, sitemaps or settings in Search Console or Bing Webmaster Tools, including the Search generative AI control.
- Editing the Business Profile or Merchant Center, or replying to reviews.
- Sending outreach to third-party sites, or spending on tools, listings or ads.
- Accepting terms or changing billing for any tool.

**Setup advice:**
- Give the agent read access to Search Console and Bing Webmaster Tools, not owner access, and a separate limited user (or exported files) for the website, CMS and CDN, never the owner’s own login.
- Review every robots.txt change before it goes live; one wrong line can block search.

## Worked example

The figures below are illustrative. Never reuse them: use only the user’s own numbers and facts, and label anything they have not given as an ASSUMPTION.

**Scenario:** The figures below are an illustration; use only the user’s own facts. A family-run boutique hotel in Porto is never named when travellers ask AI tools for hotels in the old town.

**Input:** 24 buyer questions, each run 3 times in ChatGPT, Google AI Mode, Perplexity and Copilot (288 runs; AI Overviews rarely showed for these questions). The hotel is named in 12 runs, all in Google AI Mode. robots.txt allows all bots, but the CDN has a "block AI bots" rule on. Search Console’s Search generative AI control is on Include; the site is not verified in Bing Webmaster Tools. The rooms page describes rooms only in photos; no page mentions the quiet location or distance to Ribeira. Two booking sites and a travel blog are cited most often.

**Output excerpt:** Baseline: named in 12 of 288 runs (4%), all in Google AI Mode (12 of 72 AI Mode runs, 17%). Search Console: the Search generative AI control is on Include, so Google eligibility is not the gap; check the Generative AI performance report for which pages get impressions. Access (assumption to confirm in the CDN dashboard): the "block AI bots" rule challenges OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User, which would explain the absence from ChatGPT and Perplexity. Copilot depends on Bing’s index: verify the site in Bing Webmaster Tools to see whether the key pages are indexed (verification itself does not index anything); allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and, if you want to opt out of training, block GPTBot and ClaudeBot instead. Pages: add text to the rooms and location pages that answers the top questions ("a quiet 12-room hotel a 5-minute walk from Ribeira, rooms from EUR 140"), keep prices current, and make sure the structured data matches the text. Off-site: the travel blog cited most often lists hotels by neighbourhood; send the owner accurate details and photos; ask recent guests for reviews without offering anything in return; claim and update Bing Places. Bing: verify the site in Bing Webmaster Tools, submit the changed pages with IndexNow, and watch the AI Performance report and Citation Share. Re-run the same 288 tests next month and compare.

## References

### Platform settings

| Platform | Path | Setting | Note | Source |
|---|---|---|---|---|
| robots.txt | https://yourdomain/robots.txt, user-agent group OAI-SearchBot | Allow, to be eligible for ChatGPT search answers (GPTBot can be set separately for training). | - | [t1-oai-bots] |
| robots.txt | https://yourdomain/robots.txt, user-agent group Claude-SearchBot | Allow, to be indexed for Claude search (ClaudeBot can be set separately for training). | - | [t1-ant-bots] |
| robots.txt and firewall | robots.txt user-agent group PerplexityBot; firewall allow rule for Perplexity IP ranges | Allow PerplexityBot and Perplexity’s published IP ranges. | Perplexity documents an example Cloudflare WAF rule matching the user agent and IP ranges. | [t1-pplx-bots] |
| robots.txt | https://yourdomain/robots.txt, user-agent group Google-Extended | Optional: disallow to opt out of Gemini training and grounding; has no effect on Google Search or AI Overviews. | - | [t1-g-crawlers] |
| Bing Webmaster Tools | AI Performance | Review total citations, cited pages and grounding queries monthly. | Preview since February 2026; Intents, Topics, Citation Share and Compare added in June 2026. Figures are citations, not rankings. | [t1-bing-aiperf], [t1-bing-jun] |
| Google Search Console | Settings > Search generative AI | Include (the default) to be eligible for AI Overviews, AI Mode and Discover generative AI features; Exclude removes the site from them. | - | [t1-gsc-genai-control] |
| Google Search Console | Generative AI performance report (and Performance > Search results, search type Web) | Review AI Overviews and AI Mode impressions by page, country and device monthly. | Available to all sites since 31 August 2026; no query data. | [t1-gsc-genai-report] |

### Sources

| ID | Title | Tier | Accessed on | URL |
|---|---|---|---|---|
| t1-g-ai-features | AI features and your website (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/appearance/ai-features |
| t1-g-aiopt | Optimizing your website for generative AI features on Google Search (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/fundamentals/ai-optimization-guide |
| t1-gsc-genai-control | Search generative AI control (Search Console Help) | T1 | 2026-10-29 | https://support.google.com/webmasters/answer/16908024 |
| t1-gsc-genai-report | Generative AI performance report (Search Console Help) | T1 | 2026-10-29 | https://support.google.com/webmasters/answer/16984139 |
| t1-g-usertrig | Google user-triggered fetchers, including Google-Agent (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers |
| t1-gbp-reviews | Prohibited and restricted content: fake engagement and review incentives (Google Business Profile Help) | T1 | 2026-10-29 | https://support.google.com/business/answer/3474122 |
| t1-g-robots-meta | Robots meta tag, data-nosnippet and X-Robots-Tag specifications (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag |
| t1-g-crawlers | Google’s common crawlers, including Google-Extended (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| t1-g-spam | Spam policies for Google web search: scaled content abuse (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/essentials/spam-policies |
| t1-g-genai | Google Search’s guidance on using generative AI content on your website (Google Search Central) | T1 | 2026-10-29 | https://developers.google.com/search/docs/fundamentals/using-gen-ai-content |
| t1-gbp-guidelines | Guidelines for representing your business on Google (Google Business Profile Help) | T1 | 2026-10-29 | https://support.google.com/business/answer/3038177 |
| t1-oai-bots | Overview of OpenAI crawlers (OpenAI) | T1 | 2026-10-29 | https://platform.openai.com/docs/bots |
| t1-ant-bots | Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic) | T1 | 2026-10-29 | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| t1-pplx-bots | Perplexity crawlers (Perplexity) | T1 | 2026-10-29 | https://docs.perplexity.ai/guides/bots |
| t1-bing-guidelines | Bing Webmaster Guidelines (Microsoft Bing) | T1 | 2026-10-29 | https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a |
| t1-bing-jun | New AI visibility insights in Bing Webmaster Tools: Intents, Topics, Citation Share, Compare (Bing Blogs, 16 June 2026) | T1 | 2026-10-29 | https://blogs.bing.com/search/2026/6/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare/ |
| t1-bing-oct | Introducing Topics Portfolio (beta) in AI Performance (Bing Webmaster Blog, 13 October 2026) | T1 | 2026-10-29 | https://blogs.bing.com/webmaster/2026/10/introducing-topics-portfolio-beta-in-ai-performance/ |
| t1-bing-aiperf | Introducing AI Performance in Bing Webmaster Tools, public preview (Bing Webmaster Blog, 10 February 2026) | T1 | 2026-10-29 | https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview |

---
Deepstory methods are process guidance for AI tools. Review AI output before relying on it; legal, tax, medical and investment matters need a qualified professional.
