Set it up
Turn on tracking
Pick where your site runs
Add the code and deploy
Check it works
Which option to pick
Tracking runs where your website is served, which isn’t always where you publish. A Strapi or headless site is tracked on the frontend that renders the pages, not in the CMS.By publishing connector
By framework and host
webflow.io, framer.website and
notion.site aren’t your domain, so they can’t be tracked either.
Set up each option
Next.js (own server)
- Run
npm install @seosorted/ai-crawl. - Create
instrumentation.tsat your project root, or insrc/if your app lives there. - Paste the code from the setup screen and deploy.
- If you already track crawlers in
proxy.tsormiddleware.ts, remove that call, or each visit is counted twice. - If you already have an
instrumentation.ts, add the code inside your existingregister()function instead of creating a second file. - Behind a load balancer or ingress, forward
x-forwarded-forso we get the crawler’s IP.
Next.js on Vercel
- Run
npm install @seosorted/ai-crawl. - Create
proxy.tsat your project root. On Next.js 15 and older, name itmiddleware.ts. - Paste the code from the setup screen and deploy.
proxy.ts runs on every request, and it runs before the page renders, so it can’t
see whether the page answered “not found”. Visits are still tracked, and the status shows
Missing pages can’t be found. To get both, put your domain behind Cloudflare and use the
Cloudflare option instead. Use one option, not both, or each visit is counted twice.
- If you already have a
proxy.tsormiddleware.ts, calltrackAICrawlerRequest(request, event, …)at the top of your existing function and keep your own return value. - If your
matcherdoesn’t include a path, visits to it aren’t tracked. Keep page routes in it.
Cloudflare Pages
- Run
npm install @seosorted/ai-crawl. - Create
functions/_middleware.ts, paste the code from the setup screen and deploy.
functions/_middleware.ts, export an array so both run:
export const onRequest = [yourMiddleware, seosortedMiddleware];.
Express and Node
- Run
npm install @seosorted/ai-crawl. - In your server entry file, add the code before your routes, so it sees every request.
- Deploy.
app.set("trust proxy", true) when you run behind a load balancer or proxy. Without it we
see your proxy’s IP and leave the visits out as unverified.
Fastify, Koa, NestJS (on Fastify) and other Node frameworks: use
Any other server (API) from a hook that runs after the response.
WordPress
- Connect to your site with SFTP or your host’s file manager.
- Create the
wp-content/mu-pluginsfolder if it’s missing. - Create
seosorted-ai-crawl.phpinside it and paste the code from the setup screen.
- Caching plugins (WP Rocket, LiteSpeed Cache, W3 Total Cache) and host page caches serve cached pages without running PHP, so some visits aren’t sent. Missing pages still show up, because “not found” pages aren’t cached. Use the Cloudflare option for full coverage.
- Managed hosts (WP Engine, Kinsta, Pressable) allow
mu-plugins, but their server cache skips PHP the same way. - WordPress.com gives SFTP access only on the Business and Commerce plans. Turn it on in Settings → Hosting Configuration, then upload the file. On other plans you can’t add the file, so tracking isn’t available.
Any other server (API)
Send each crawler request to your tracking URL from the server, after the response is sent so it never slows a page down:- Send up to 500 hits per request, and at most 1,200 requests a minute per website ID.
- Send only crawler requests and skip asset files (CSS, JS, images, fonts). The setup screen’s Cloudflare and WordPress code show the user-agent pattern we use.
- Ignore errors and don’t retry inside the request. Tracking must never break your site.
Cloudflare (any site)
A Cloudflare Worker runs in front of your site, sees every request, and reports crawler visits to SEO Sorted. It works with any host whose domain is proxied through Cloudflare (orange cloud). For Webflow and Framer, follow Webflow or Framer instead: they need platform-specific DNS records.Put your domain on Cloudflare
- Create a free Cloudflare account, choose Onboard a domain and enter your root domain.
- Check the imported DNS records against the ones your host gives you.
- At your domain registrar (GoDaddy, Namecheap and so on), replace the nameservers with the two Cloudflare shows you. The change can take up to 48 hours. The domain shows Active on your Cloudflare overview once it’s done.
Proxy the records that serve your site
- In Cloudflare → DNS, set the root and
wwwrecords to Proxied (orange cloud). - In SSL/TLS → Overview, use Full or Full (strict). Flexible causes a redirect loop on hosts that force HTTPS.
- Open your site in a browser and confirm it still loads.
Copy the tracking code from SEO Sorted
Create a Cloudflare Worker
- In Cloudflare, go to Workers & Pages → Create → Create Worker, name it (for example
seosorted-ai-crawlers) and click Deploy. - Click Edit code, replace everything with the code from SEO Sorted, and click Deploy again.
Attach the Worker to your domain
Check it works
Webflow
Webflow supports Cloudflare in front of its hosting through Cloudflare’s Orange-to-Orange (O2O) setup (Webflow’s guide). It works on any Cloudflare plan, including Free. Nothing changes inside your Webflow project. You need:- A paid Webflow Site plan and a custom domain.
- A site on Webflow’s current hosting. Sites created after 21 April 2025 already are. For an older
site, the domain must use Webflow’s current DNS records (below), not the old two A records or
proxy-ssl.webflow.com.
Put your domain on Cloudflare
Point the domain at Webflow with proxied CNAME records
proxy-ssl.webflow.com record,
then add:blog.Connect and publish the domain in Webflow
Add the SEO Sorted Worker
Framer
Framer supports Cloudflare as a reverse proxy on Pro with the Advanced Hosting add-on and on Enterprise (Framer’s guide). On other plans, tracking isn’t available.Set up Framer
your-site.framer.website). At the bottom of that page, enter your custom domain as the
canonical URL under Advanced, then publish.Put your domain on Cloudflare
@ (and www if you use it) pointing to the placeholder
address 192.0.2.1. The Worker answers every request, so the address is never reached.Create the Worker and point it at Framer
const res = await fetch(request);, so it fetches from your Framer subdomain:your-site.com/* (and www.your-site.com/*).Turn off Rocket Loader and test
Your setup is different?
My domain is already on Cloudflare
My domain is already on Cloudflare
cdn.webflow.com CNAMEs above.Only a subdomain runs on the platform (for example blog.your-site.com)
Only a subdomain runs on the platform (for example blog.your-site.com)
blog.your-site.com/*. The rest
of your domain is untouched. If the main site runs on a platform with its own option (Next.js,
WordPress and so on), track it with that option. Both report to the same website ID.My site only uses www, or only the root domain
My site only uses www, or only the root domain
My site is on a webflow.io or framer.website address
My site is on a webflow.io or framer.website address
I already have a Worker on my domain
I already have a Worker on my domain
My registrar won't let me change nameservers
My registrar won't let me change nameservers
Wix, Squarespace, Shopify or Ghost(Pro)
Wix, Squarespace, Shopify or Ghost(Pro)
HubSpot CMS
HubSpot CMS
What the status means
Troubleshooting
The status stays on “Waiting”
The status stays on “Waiting”
- Check the change is deployed to the live site, not only saved.
- Check the website ID in the code matches the one on the setup screen. Rotating the key stops the old ID immediately.
- A normal browser visit isn’t sent. Use the test command, which pretends to be a crawler.
- Run the second test command under Still waiting? Troubleshoot. It skips your site and calls SEO Sorted directly. If that connects but the site test doesn’t, the code isn’t running on your site yet.
“Most AI crawler visits come from addresses that don't belong to the crawler”
“Most AI crawler visits come from addresses that don't belong to the crawler”
app.set("trust proxy", true). Elsewhere, forward the x-forwarded-for header.A page I fixed still shows as missing
A page I fixed still shows as missing
My site shows a page for every address
My site shows a page for every address
Cloudflare: the status stays on “Waiting”
Cloudflare: the status stays on “Waiting”
- The DNS record must be Proxied (orange cloud). With DNS only, requests go straight to your host and the Worker never runs.
- The Worker needs a route for the exact hostname you tested:
your-site.com/*andwww.your-site.com/*are separate routes. - Check the nameserver change has finished: the domain shows Active on your Cloudflare overview.
- Click Deploy after pasting the code. Saving a draft doesn’t put it live.
Cloudflare: “Too many redirects” after proxying
Cloudflare: “Too many redirects” after proxying
Webflow: “525 Handshake Error”
Webflow: “525 Handshake Error”
proxy-ssl.webflow.com). Replace them with the proxied cdn.webflow.com CNAMEs in
Webflow. If you can’t, set the records to DNS only, which turns tracking off.Webflow: “Update needed” next to the domain
Webflow: “Update needed” next to the domain
WordPress with a caching plugin
WordPress with a caching plugin

