> ## Documentation Index
> Fetch the complete documentation index at: https://faii.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Systems Tracked

> Complete list of AI crawlers and bots that FAII identifies and monitors when they visit your website.

<Info>
  **Quick Answer:** FAII tracks 6+ AI crawler systems including OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, and Google's AI crawlers. Each has different crawling patterns and purposes.
</Info>

## What You'll Learn

* Which AI bots FAII identifies
* How each bot behaves and what it's looking for
* How to interpret visits from different bots
* How to ensure your site is accessible to the right bots

## AI Crawlers Overview

| Bot                 | Owner      | Purpose                       | Crawl Frequency |
| ------------------- | ---------- | ----------------------------- | :-------------: |
| **GPTBot**          | OpenAI     | Training data & web browsing  |   Daily-Weekly  |
| **ChatGPT-User**    | OpenAI     | Real-time web access in chats |    On-demand    |
| **ClaudeBot**       | Anthropic  | Training data collection      |  Weekly-Monthly |
| **PerplexityBot**   | Perplexity | Real-time search answers      |     Frequent    |
| **Google-Extended** | Google     | Gemini AI training            |      Daily      |
| **Googlebot**       | Google     | Search index + AI Overviews   |  Very frequent  |

## Detailed Bot Profiles

### GPTBot (OpenAI)

| Property         | Details                                                     |
| ---------------- | ----------------------------------------------------------- |
| **User-Agent**   | `Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.0)`   |
| **Purpose**      | Crawls web pages for ChatGPT's training and knowledge       |
| **Behavior**     | Systematic crawling, respects robots.txt                    |
| **Significance** | Visits suggest your content may appear in ChatGPT responses |

**What GPTBot visits mean:** OpenAI is actively reading your content. Pages visited by GPTBot are candidates for inclusion in ChatGPT's knowledge base.

### ChatGPT-User (OpenAI)

| Property         | Details                                                         |
| ---------------- | --------------------------------------------------------------- |
| **User-Agent**   | `Mozilla/5.0 AppleWebKit/537.36 (compatible; ChatGPT-User/1.0)` |
| **Purpose**      | Real-time web access during user conversations                  |
| **Behavior**     | On-demand visits triggered by user queries                      |
| **Significance** | Someone asked ChatGPT a question that led to your page          |

**What ChatGPT-User visits mean:** A real user asked ChatGPT something, and ChatGPT browsed to your page for the answer. This is the strongest signal - your content is actively being used in responses.

### ClaudeBot (Anthropic)

| Property         | Details                                           |
| ---------------- | ------------------------------------------------- |
| **User-Agent**   | `ClaudeBot/1.0`                                   |
| **Purpose**      | Web content collection for Claude's knowledge     |
| **Behavior**     | Periodic crawling, less frequent than GPTBot      |
| **Significance** | Your content is being read by Anthropic's systems |

**What ClaudeBot visits mean:** Anthropic is indexing your content for Claude's knowledge. Pages visited are more likely to be referenced in Claude's responses.

### PerplexityBot (Perplexity)

| Property         | Details                                                      |
| ---------------- | ------------------------------------------------------------ |
| **User-Agent**   | `PerplexityBot`                                              |
| **Purpose**      | Real-time search and citation                                |
| **Behavior**     | Very frequent, search-driven crawling                        |
| **Significance** | Your content is being used as a source in Perplexity answers |

**What PerplexityBot visits mean:** Perplexity actively cites sources in its answers. Visits mean your pages are being considered as citation sources - the most direct path to appearing in Perplexity responses.

### Google-Extended (Google)

| Property         | Details                                      |
| ---------------- | -------------------------------------------- |
| **User-Agent**   | `Google-Extended`                            |
| **Purpose**      | AI/ML training (Gemini, Bard successors)     |
| **Behavior**     | Follows Googlebot patterns                   |
| **Significance** | Your content may inform Google's AI products |

**What Google-Extended visits mean:** Google is reading your content specifically for AI training purposes (separate from search indexing). This content may influence Gemini's responses.

### Googlebot (Google)

| Property         | Details                                                      |
| ---------------- | ------------------------------------------------------------ |
| **User-Agent**   | `Googlebot/2.1`                                              |
| **Purpose**      | Search indexing + AI Overview sources                        |
| **Behavior**     | Very frequent, comprehensive                                 |
| **Significance** | Required for both organic rankings and AI Overview citations |

**What Googlebot visits mean:** Standard search indexing, but also feeds into AI Overview source selection. Pages indexed by Googlebot are candidates for AI Overview citations.

## Interpreting Bot Activity

### High-value patterns

| Pattern                          | What It Suggests                              |
| -------------------------------- | --------------------------------------------- |
| Multiple bots visiting same page | Page is broadly discoverable and relevant     |
| ChatGPT-User visits              | Real users are being directed to your content |
| PerplexityBot frequent visits    | Your content is actively cited in answers     |
| Increasing visit frequency       | Growing authority in the AI ecosystem         |

### Concerning patterns

| Pattern                    | What It Suggests                       |
| -------------------------- | -------------------------------------- |
| No visits from any bot     | Site may be blocked or undiscoverable  |
| Visits stopped suddenly    | Check robots.txt for accidental blocks |
| Only Googlebot, no AI bots | AI-specific bots may be blocked        |
| 404 responses to bots      | Broken pages need fixing               |

## Managing Bot Access

### robots.txt configuration

Your `robots.txt` file controls which bots can access your site:

```
# Allow all AI bots (recommended for AI visibility)
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /
```

<Warning>
  Blocking AI bots in robots.txt means they cannot read your content, which means they cannot recommend your brand. Only block bots if you have specific content licensing or privacy concerns.
</Warning>

### Selective access

If you want AI bots to read some pages but not others:

```
User-agent: GPTBot
Allow: /blog/
Allow: /services/
Disallow: /internal/
Disallow: /drafts/
```

## New Bots

FAII continuously updates its bot detection database as new AI crawlers emerge. When a new AI platform launches a crawler, FAII adds detection automatically through plugin updates.

Currently monitoring for upcoming crawlers from:

* Mistral AI
* Meta AI
* Apple (Siri/Apple Intelligence)

## FAQ

<AccordionGroup>
  <Accordion title="What if a bot visits but I'm not getting mentioned?">
    Bot visits are a prerequisite, not a guarantee. The bot reading your content doesn't automatically mean the AI will recommend you. Your content also needs to be relevant, authoritative, and well-structured for the topics users ask about.
  </Accordion>

  <Accordion title="Can I attract more bot visits?">
    Publish fresh, valuable content regularly. Submit your sitemap to search engines. Build quality backlinks. AI bots tend to follow links from authoritative sites and prioritize frequently-updated content.
  </Accordion>

  <Accordion title="Do bot visits cost me bandwidth?">
    AI bot visits are typically lightweight (they mostly read HTML content) and infrequent enough to have negligible bandwidth impact. Most sites see fewer than 1,000 bot visits per month.
  </Accordion>

  <Accordion title="How do I know if a visit led to a mention?">
    FAII correlates bot visits with subsequent Chat Intelligence data. If a bot visits your page and you later get mentioned for that topic, the correlation appears in your analytics.
  </Accordion>
</AccordionGroup>

## Related Articles

<CardGroup cols={2}>
  <Card title="How Bot Tracking Works" icon="robot" href="/docs/ai-website-analytics/how-bot-tracking-works">
    Technical detection overview
  </Card>

  <Card title="Most Visited Pages" icon="file-lines" href="/docs/ai-website-analytics/most-visited-pages">
    Which pages bots visit most
  </Card>
</CardGroup>