Skip to main content
AI crawlers don’t run JavaScript, so an analytics tag never sees them. Crawler tracking runs on your server or CDN instead: a few lines of code send each crawler request to SEO Sorted. It powers the AI crawler visits number on your dashboard and the Pages AI crawler can’t open card on the AI Visibility page.
Only crawler requests are sent. Visits from people are never sent, and the code never slows your pages down: it sends in the background and ignores errors.

Set it up

1

Turn on tracking

Open Project Settings → AI Crawler Tracking (or Set up tracking on the AI Visibility card) and click Turn on tracking. You need to be a workspace admin.
2

Pick where your site runs

Choose your platform. The code shown already contains your website ID.
3

Add the code and deploy

Paste it where the screen says, then deploy. If someone else manages your site, use Copy instructions for your developer and send them the text.
4

Check it works

Run the test command from the setup screen:
The status turns to Connected within a few seconds. Test requests never count as crawler visits.

Which option to pick

Tracking runs where your website is served, which isn’t always where you publish. A Strapi or headless site is tracked on the frontend that renders the pages, not in the CMS.
If your domain already runs through Cloudflare (orange cloud), the Cloudflare (any site) option works for every platform below and always finds missing pages.

By publishing connector

By framework and host

Hosted platforms don’t allow server code, so tracking runs in a Cloudflare Worker in front of them. That only works where the platform supports a Cloudflare-proxied domain: Webflow (through Cloudflare’s Orange-to-Orange setup) and Framer (through a reverse proxy on eligible plans). Wix, Shopify and Ghost(Pro) require DNS-only records, and Squarespace recommends them, so tracking isn’t available there. Platform addresses such as webflow.io, framer.website and notion.site aren’t your domain, so they can’t be tracked either.

Set up each option

Next.js (own server)

  1. Run npm install @seosorted/ai-crawl.
  2. Create instrumentation.ts at your project root, or in src/ if your app lives there.
  3. Paste the code from the setup screen and deploy.
It reads each response after it’s sent, so every visit carries the real status code, including 404s served from Next.js’s cache and redirects.
  • If you already track crawlers in proxy.ts or middleware.ts, remove that call, or each visit is counted twice.
  • If you already have an instrumentation.ts, add the code inside your existing register() function instead of creating a second file.
  • Behind a load balancer or ingress, forward x-forwarded-for so we get the crawler’s IP.

Next.js on Vercel

  1. Run npm install @seosorted/ai-crawl.
  2. Create proxy.ts at your project root. On Next.js 15 and older, name it middleware.ts.
  3. Paste the code from the setup screen and deploy.
On Vercel only proxy.ts runs on every request, and it runs before the page renders, so it can’t see whether the page answered “not found”. Visits are still tracked, and the status shows Missing pages can’t be found. To get both, put your domain behind Cloudflare and use the Cloudflare option instead. Use one option, not both, or each visit is counted twice.
  • If you already have a proxy.ts or middleware.ts, call trackAICrawlerRequest(request, event, …) at the top of your existing function and keep your own return value.
  • If your matcher doesn’t include a path, visits to it aren’t tracked. Keep page routes in it.

Cloudflare Pages

  1. Run npm install @seosorted/ai-crawl.
  2. Create functions/_middleware.ts, paste the code from the setup screen and deploy.
If you already have functions/_middleware.ts, export an array so both run: export const onRequest = [yourMiddleware, seosortedMiddleware];.

Express and Node

  1. Run npm install @seosorted/ai-crawl.
  2. In your server entry file, add the code before your routes, so it sees every request.
  3. Deploy.
Keep app.set("trust proxy", true) when you run behind a load balancer or proxy. Without it we see your proxy’s IP and leave the visits out as unverified. Fastify, Koa, NestJS (on Fastify) and other Node frameworks: use Any other server (API) from a hook that runs after the response.

WordPress

  1. Connect to your site with SFTP or your host’s file manager.
  2. Create the wp-content/mu-plugins folder if it’s missing.
  3. Create seosorted-ai-crawl.php inside it and paste the code from the setup screen.
Must-use plugins load automatically, with nothing to activate, and keep working through theme changes. This also covers WooCommerce stores.
  • Caching plugins (WP Rocket, LiteSpeed Cache, W3 Total Cache) and host page caches serve cached pages without running PHP, so some visits aren’t sent. Missing pages still show up, because “not found” pages aren’t cached. Use the Cloudflare option for full coverage.
  • Managed hosts (WP Engine, Kinsta, Pressable) allow mu-plugins, but their server cache skips PHP the same way.
  • WordPress.com gives SFTP access only on the Business and Commerce plans. Turn it on in Settings → Hosting Configuration, then upload the file. On other plans you can’t add the file, so tracking isn’t available.

Any other server (API)

Send each crawler request to your tracking URL from the server, after the response is sent so it never slows a page down:
  • Send up to 500 hits per request, and at most 1,200 requests a minute per website ID.
  • Send only crawler requests and skip asset files (CSS, JS, images, fonts). The setup screen’s Cloudflare and WordPress code show the user-agent pattern we use.
  • Ignore errors and don’t retry inside the request. Tracking must never break your site.

Cloudflare (any site)

A Cloudflare Worker runs in front of your site, sees every request, and reports crawler visits to SEO Sorted. It works with any host whose domain is proxied through Cloudflare (orange cloud). For Webflow and Framer, follow Webflow or Framer instead: they need platform-specific DNS records.
1

Put your domain on Cloudflare

  1. Create a free Cloudflare account, choose Onboard a domain and enter your root domain.
  2. Check the imported DNS records against the ones your host gives you.
  3. At your domain registrar (GoDaddy, Namecheap and so on), replace the nameservers with the two Cloudflare shows you. The change can take up to 48 hours. The domain shows Active on your Cloudflare overview once it’s done.
2

Proxy the records that serve your site

  1. In Cloudflare → DNS, set the root and www records to Proxied (orange cloud).
  2. In SSL/TLS → Overview, use Full or Full (strict). Flexible causes a redirect loop on hosts that force HTTPS.
  3. Open your site in a browser and confirm it still loads.
3

Copy the tracking code from SEO Sorted

Open Project Settings → AI Crawler Tracking, click Turn on tracking, and choose Cloudflare (any site). Copy the code. It already contains your website ID.
4

Create a Cloudflare Worker

  1. In Cloudflare, go to Workers & Pages → Create → Create Worker, name it (for example seosorted-ai-crawlers) and click Deploy.
  2. Click Edit code, replace everything with the code from SEO Sorted, and click Deploy again.
5

Attach the Worker to your domain

In the Worker, open Settings → Domains & Routes → Add → Route, pick your domain, and add a route for every hostname that serves pages:
Leave the failure mode on Fail open, so your site keeps loading even if the Worker hits a limit.
6

Check it works

Run the test command from the setup screen. The status turns to Connected within a few seconds, and to Live once a real AI crawler visits, which can take a few days on smaller sites.
The Worker runs on every request to your site, including visitors and images. Cloudflare’s free Workers plan covers 100,000 requests a day. Busier sites need Workers Paid (from $5 a month). With Fail open set, requests above the limit still reach your site and are simply not tracked.

Webflow

Webflow supports Cloudflare in front of its hosting through Cloudflare’s Orange-to-Orange (O2O) setup (Webflow’s guide). It works on any Cloudflare plan, including Free. Nothing changes inside your Webflow project. You need:
  • A paid Webflow Site plan and a custom domain.
  • A site on Webflow’s current hosting. Sites created after 21 April 2025 already are. For an older site, the domain must use Webflow’s current DNS records (below), not the old two A records or proxy-ssl.webflow.com.
1

Put your domain on Cloudflare

Follow Put your domain on Cloudflare in Cloudflare (any site). If your domain is already on Cloudflare, skip this.
2

Point the domain at Webflow with proxied CNAME records

In Cloudflare → DNS, delete any old Webflow A records or proxy-ssl.webflow.com record, then add:Add the same kind of record for any other subdomain Webflow serves, such as blog.
3

Connect and publish the domain in Webflow

In Webflow → Site settings → Publishing → Production, connect the custom domain, click Verify domain, set the default domain, and publish.
Webflow may keep showing Update needed next to the domain. That’s expected with the proxy on, because Cloudflare hides your DNS records from Webflow’s check. If the site loads, nothing is wrong.
4

Add the SEO Sorted Worker

Follow steps 3 to 6 of Cloudflare (any site): copy the code, create the Worker, add a route for each hostname, and run the test.
If your domain has CAA records, they must allow Let’s Encrypt and Google Trust Services, or Webflow can’t issue or renew your SSL certificate. If you use Cloudflare’s Zone Hold, release it while Webflow issues the certificate.

Framer

Framer supports Cloudflare as a reverse proxy on Pro with the Advanced Hosting add-on and on Enterprise (Framer’s guide). On other plans, tracking isn’t available.
1

Set up Framer

In Site Settings → Hosting → Domains, add a free Framer subdomain (for example your-site.framer.website). At the bottom of that page, enter your custom domain as the canonical URL under Advanced, then publish.
2

Put your domain on Cloudflare

Follow Put your domain on Cloudflare in Cloudflare (any site). Then add a Proxied A record for @ (and www if you use it) pointing to the placeholder address 192.0.2.1. The Worker answers every request, so the address is never reached.
3

Create the Worker and point it at Framer

Create the Worker with the SEO Sorted code, as in Cloudflare (any site). Then change its first line, const res = await fetch(request);, so it fetches from your Framer subdomain:
Deploy, and add the routes your-site.com/* (and www.your-site.com/*).
4

Turn off Rocket Loader and test

Framer asks you to turn off Rocket Loader (Speed → Optimization → Content optimizations). Then run the test command from the setup screen.

Your setup is different?

Skip Put your domain on Cloudflare. Check the records serving your site are Proxied and SSL/TLS is Full or Full (strict), then add the Worker. For Webflow, make sure the records are the cdn.webflow.com CNAMEs above.
Proxy only that subdomain’s record, and add one route for it: blog.your-site.com/*. The rest of your domain is untouched. If the main site runs on a platform with its own option (Next.js, WordPress and so on), track it with that option. Both report to the same website ID.
Add a route for every hostname that serves pages. Add the other one too if it redirects: crawlers that follow the redirect then show up on both. A missing route is the most common reason visits never arrive.
Those addresses belong to the platform, so you can’t put them behind your Cloudflare account. Connect a custom domain first.
Cloudflare runs only one Worker per request, the one on the most specific matching route. Instead of adding a second Worker, add the SEO Sorted code’s logic to your existing Worker: after it gets the response, send the crawler visit the same way, then return the response.
Cloudflare’s CNAME (partial) setup avoids changing nameservers, but it needs the Cloudflare Business plan. Otherwise, move the domain to a registrar that allows custom nameservers, or to Cloudflare Registrar.
These platforms don’t support a Cloudflare-proxied domain. Wix, Shopify and Ghost(Pro) require DNS-only records, and Squarespace recommends them. Proxying anyway breaks SSL issuance or renewal and can take the site offline, so AI crawler tracking isn’t available on them.
HubSpot supports Cloudflare in front of its hosting only through an Orange-to-Orange setup on a Cloudflare Enterprise plan. With Enterprise, follow HubSpot’s reverse proxy guide, then add the Worker as in Cloudflare (any site).

What the status means

Troubleshooting

  • Check the change is deployed to the live site, not only saved.
  • Check the website ID in the code matches the one on the setup screen. Rotating the key stops the old ID immediately.
  • A normal browser visit isn’t sent. Use the test command, which pretends to be a crawler.
  • Run the second test command under Still waiting? Troubleshoot. It skips your site and calls SEO Sorted directly. If that connects but the site test doesn’t, the code isn’t running on your site yet.
We check each visit claiming to be GPTBot, ChatGPT, Perplexity or Googlebot against the crawler’s published addresses, and leave fakes out. If your server sits behind a load balancer or proxy, it may be sending its own address instead of the visitor’s. On Express, keep app.set("trust proxy", true). Elsewhere, forward the x-forwarded-for header.
The card re-checks every page live before listing it, so a redirect you’ve added drops it from the list on the next check. Open the card’s details and click Re-check.
Some sites answer every unknown address with a normal page instead of “not found” (a soft 404). Crawlers then never get a “not found”, so nothing can be listed. Make unknown addresses return a real 404 status.
  • The DNS record must be Proxied (orange cloud). With DNS only, requests go straight to your host and the Worker never runs.
  • The Worker needs a route for the exact hostname you tested: your-site.com/* and www.your-site.com/* are separate routes.
  • Check the nameserver change has finished: the domain shows Active on your Cloudflare overview.
  • Click Deploy after pasting the code. Saving a draft doesn’t put it live.
SSL/TLS is set to Flexible. Change it to Full or Full (strict) in SSL/TLS → Overview.
The domain is proxied but still uses Webflow’s old records (two A records or proxy-ssl.webflow.com). Replace them with the proxied cdn.webflow.com CNAMEs in Webflow. If you can’t, set the records to DNS only, which turns tracking off.
Expected with the proxy on. Cloudflare hides your DNS records from Webflow’s check. If the site loads and publishes, nothing needs fixing.
WP Rocket, LiteSpeed Cache and similar serve cached pages without running PHP, so some visits aren’t sent. Missing pages still show up, because “not found” pages aren’t cached. For full coverage, use the Cloudflare option.

Turning it off

Turn off tracking on the setup screen stops SEO Sorted accepting data from your site. What was recorded stays, and turning it back on resumes with the same website ID. You can then remove the code from your site.