> ## Documentation Index
> Fetch the complete documentation index at: https://docs.seosorted.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawler tracking

> See which pages GPTBot, ClaudeBot, ChatGPT and other AI crawlers visit, and which ones they can't open.

AI crawlers don't run JavaScript, so an analytics tag never sees them. Crawler tracking runs on
your server or CDN instead: a few lines of code send each crawler request to SeoSorted. It powers
the **AI crawler visits** number on your dashboard and the **Pages AI crawler can't open** card on
the AI Visibility page.

<Note>
  Only crawler requests are sent. Visits from people are never sent, and the code never slows
  your pages down: it sends in the background and ignores errors.
</Note>

## Set it up

<Steps>
  <Step title="Turn on tracking">
    Open **Project Settings → AI Crawler Tracking** (or **Set up tracking** on the AI Visibility
    card) and click **Turn on tracking**. You need to be a workspace admin.
  </Step>

  <Step title="Pick where your site runs">
    Choose your platform. The code shown already contains your website ID.
  </Step>

  <Step title="Add the code and deploy">
    Paste it where the screen says, then deploy. If someone else manages your site, use
    **Copy instructions for your developer** and send them the text.
  </Step>

  <Step title="Check it works">
    Run the test command from the setup screen:

    ```bash theme={null}
    curl -A "SEOSortedTest/1.0" https://your-site.com/
    ```

    The status turns to **Connected** within a few seconds. Test requests never count as
    crawler visits.
  </Step>
</Steps>

## Which option to pick

| Your site | Option | Finds missing pages? |
| - | - | - |
| Next.js, on Vercel or self-hosted | Next.js / Vercel | No, visits only |
| Any site whose domain runs through Cloudflare | Cloudflare (any site) | Yes |
| Cloudflare Pages | Cloudflare Pages | Yes |
| Express or another Node server | Express / Node | Yes |
| Self-hosted WordPress | WordPress | Yes |
| Anything else with server code | Any other server (API) | Yes, if you send the status code |
| Shopify, Wix, Squarespace, Webflow, Framer | Only through Cloudflare | Yes, via Cloudflare |

**Why Next.js can't find missing pages:** `proxy.ts` runs before the page renders, so it can't
see whether the page answered "not found". Visits are still tracked. If your domain runs through
Cloudflare, use the Cloudflare option to get both.

**Hosted website builders** (Shopify, Wix, Squarespace, Webflow, Framer) don't allow server code.
They work only if your domain is proxied through Cloudflare (orange cloud), where the Cloudflare
option runs in front of them. Shopify's storefront can't be proxied this way.

## What the status means

| Status | Meaning |
| - | - |
| Waiting for the first visit | Tracking is on and nothing has arrived. Deploy the code and run the test command. |
| Connected | The test request arrived, so your setup works. Crawler visits appear as crawlers come by, which can take a few days on small sites. |
| Tracking works | Crawler visits arrive, but no AI crawler has visited in the last 30 days. |
| Missing pages can't be found | AI visits arrive without status codes (usually the Next.js option). Switch to an option that finds missing pages. |
| Live | Everything works. |
| Stopped receiving data | Data used to arrive and hasn't for 7 days. The code was probably removed, or the website ID changed. |

## Troubleshooting

<AccordionGroup>
  <Accordion title="The status stays on “Waiting”">
    * Check the change is deployed to the live site, not only saved.
    * Check the website ID in the code matches the one on the setup screen. Rotating the key
      stops the old ID immediately.
    * A normal browser visit isn't sent. Use the test command, which pretends to be a crawler.
    * Run the second test command under **Still waiting? Troubleshoot**. It skips your site and
      calls SeoSorted directly. If that connects but the site test doesn't, the code isn't
      running on your site yet.
  </Accordion>

  <Accordion title="“Most AI crawler visits come from addresses that don't belong to the crawler”">
    We check each visit claiming to be GPTBot, ChatGPT, Perplexity or Googlebot against the
    crawler's published addresses, and leave fakes out. If your server sits behind a load balancer
    or proxy, it may be sending its own address instead of the visitor's. On Express, keep
    `app.set("trust proxy", true)`. Elsewhere, forward the `x-forwarded-for` header.
  </Accordion>

  <Accordion title="A page I fixed still shows as missing">
    The card re-checks every page live before listing it, so a redirect you've added drops it
    from the list on the next check. Open the card's details and click **Re-check**.
  </Accordion>

  <Accordion title="My site shows a page for every address">
    Some sites answer every unknown address with a normal page instead of “not found” (a soft
    404\). Crawlers then never get a “not found”, so nothing can be listed. Make unknown
    addresses return a real 404 status.
  </Accordion>

  <Accordion title="WordPress with a caching plugin">
    WP Rocket, LiteSpeed Cache and similar serve cached pages without running PHP, so some
    visits aren't sent. Missing pages still show up, because “not found” pages aren't cached.
    For full coverage, use the Cloudflare option.
  </Accordion>
</AccordionGroup>

## Turning it off

**Turn off tracking** on the setup screen stops SeoSorted accepting data from your site. What was
recorded stays, and turning it back on resumes with the same website ID. You can then remove the
code from your site.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.