Measuring AI Search Visibility in 2026: Crawlers, GA4 and Logs

1. The Problem: Your Dashboard Measures the Wrong Things

AI search breaks measurement in three places, and each one fails in a different direction. Bots that run your JavaScript inflate Direct with fake sessions. Bots that only fetch HTML never reach GA4 at all, so you never see them. And real people who click an answer inside an AI app often arrive with no referrer, so they hide in Direct too. One dashboard cannot untangle that: you need three data sources, analytics, server logs and robots.txt.

The comparison that matters: about 17 human sessions from classic search for every one from an AI assistant, here. With numbers this small I would not read a trend into it, but I would not trust a report that shows AI traffic as zero either, or one that shows Direct as 91% of my audience.

2. Minimum Concepts Before Touching Code

Six ideas cover almost everything you need. With these straight, the robots.txt, the GA4 channel and the log script read like plain English.

3. Step 1 — Inventory the Bots (From Vendor Docs)

Start from what vendors publish about themselves, not from blog posts: names change and some guides still list agents the vendor no longer documents (Anthropic’s current help article lists ClaudeBot, Claude-User and Claude-SearchBot and does not mention anthropic-ai). Everything below was checked against each vendor’s own page in October 2026.

4. Step 2 — robots.txt, One Group per Job

robots.txt is where the taxonomy becomes policy: allow the bots that retrieve your content for answers, decide training separately, and keep one catch-all group. Several User-agent lines can sit above one set of rules, as Google’s own docs show.

public/robots.txt — one group per job (Pages Router serves it as a static file)

# 1) Search and retrieval bots: they feed answers, allow them
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
User-agent: Applebot
Allow: /
Disallow: /admin
Disallow: /api/

# 2) User-triggered fetchers: a person asked about one of your pages.
#    Some vendors say these may ignore robots.txt anyway.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: meta-externalfetcher
Allow: /
Disallow: /admin
Disallow: /api/

# 3) Training / model improvement: YOUR decision (example: opt out)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# 4) Everyone else (Googlebot, Bingbot, ...)
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

Sitemap: https://www.diego-rodriguez.work/sitemap.xml

If you generate the file with next-sitemap, the same groups go under robotsTxtOptions.policies. One subtlety bites people: Google’s docs say a user agent matches only the first, most specific group, so a bot-specific group does not inherit the Disallow lines of the * group. That is why every group above repeats /admin and /api/.

5. Step 3 — GA4: The AI Assistant Channel and a Regex Fallback

Some older guides still tell you GA4 has no AI channel and that you must build one. Google’s documentation says it does: an AI Assistant channel was added to the default channel group on May 13, 2026 (Analytics release notes). The rule is that the medium exactly matches ai-assistant, and GA4 sets that medium (and the campaign (ai-assistant)) when the referrer matches a list of AI assistants. Google names ChatGPT, Gemini, DeepSeek, Copilot and Grok as examples, publishes the full list only as a spreadsheet, and keeps AI Overviews and AI Mode in Organic Search.

Two limits matter. First, it only works when GA4 saw a recognised referrer: sessions with neither referrer nor tag still fall to Direct, and independent write-ups also report tagged-but-medium-less sessions ending in Unassigned. Second, the list is Google’s, not yours. On this property perplexity.ai already landed in AI Assistant, while one independent write-up reports Perplexity staying in Referral, so verify on your own data. A custom channel group gives you a list you control, next to the native one.

GA4 Admin → Data display → Channel groups → Create new channel group

Channel name:  AI assistants (custom)
Condition:     Source   matches regex
Pattern:       ^(.*\.)?(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.com|copilot\.microsoft\.com|deepseek\.com|grok\.com|you\.com)$
Order:         drag it ABOVE Referral (and above Organic Search if AI sources show up there)

scripts/ga4-channels.js — channels with engagement, bot-like Direct flagged

// npm i @google-analytics/data
// GOOGLE_APPLICATION_CREDENTIALS=key.json GA_PROPERTY_ID=123456789 node scripts/ga4-channels.js
const { BetaAnalyticsDataClient } = require("@google-analytics/data");

const client = new BetaAnalyticsDataClient();
const HUMAN = ["Organic Search", "AI Assistant", "Referral", "Organic Social"];

async function main() {
  const [res] = await client.runReport({
    property: "properties/" + process.env.GA_PROPERTY_ID,
    dateRanges: [{ startDate: "90daysAgo", endDate: "today" }],
    dimensions: [{ name: "sessionDefaultChannelGroup" }, { name: "sessionSource" }],
    metrics: [{ name: "sessions" }, { name: "userEngagementDuration" }],
    limit: 500,
  });

  const by = {};
  for (const row of res.rows || []) {
    const [channel, source] = row.dimensionValues.map((d) => d.value);
    const sessions = Number(row.metricValues[0].value);
    const engaged = Number(row.metricValues[1].value); // seconds, summed
    const c = (by[channel] ||= { sessions: 0, engaged: 0, sources: {} });
    c.sessions += sessions;
    c.engaged += engaged;
    c.sources[source] = (c.sources[source] || 0) + sessions;
  }

  const total = Object.values(by).reduce((n, c) => n + c.sessions, 0);
  const sorted = Object.entries(by).sort((a, b) => b[1].sessions - a[1].sessions);
  for (const [name, c] of sorted) {
    const avg = c.engaged / Math.max(c.sessions, 1);
    const top = Object.entries(c.sources).sort((a, b) => b[1] - a[1]).slice(0, 3).map(([s, n]) => s + " " + n).join(", ");
    // heuristic, not a verdict: lots of sessions with ~0 s engagement
    const verdict = HUMAN.includes(name) ? "human" : avg < 1 && c.sessions > 20 ? "BOT-LIKE" : "check";
    console.log([name.padEnd(18), String(c.sessions).padStart(6), ((100 * c.sessions) / total).toFixed(0) + "%", avg.toFixed(1) + "s", verdict, top].join("  "));
  }
}
main().catch((e) => { console.error(e.message); process.exit(1); });

Run it and read the engagement column next to the session count. A channel with hundreds of sessions and under a second of engagement is not an audience; that is the number to remove before anyone presents an AI-share percentage.

6. Step 4 — Logs: Count the Bots GA4 Cannot See

Your server or CDN logs record every request, including from clients that never run JavaScript. Two cheap ways to get a bot report: parse the access log you already have, or emit one JSON line per AI-bot hit from middleware.

scripts/ai-bots-report.js — hits per bot from a combined-format log

// node scripts/ai-bots-report.js /var/log/nginx/access.log   (Node 16+)
const fs = require("node:fs");
const readline = require("node:readline");

// user-agent token -> job
const BOTS = {
  "OAI-SearchBot": "search", GPTBot: "training", "ChatGPT-User": "user fetch",
  "Claude-SearchBot": "search", ClaudeBot: "training", "Claude-User": "user fetch",
  PerplexityBot: "search", "Perplexity-User": "user fetch",
  Applebot: "search (+training)", "meta-webindexer": "search",
  "meta-externalagent": "training", "meta-externalfetcher": "user fetch",
};

// combined log format: ip - - [time] "METHOD path HTTP/x" status bytes "referer" "user-agent"
const LINE = /^(\S+) \S+ \S+ \[[^\]]+\] "\S+ (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"/;

async function main(file) {
  const stats = {};
  const rl = readline.createInterface({ input: fs.createReadStream(file) });
  for await (const line of rl) {
    const m = LINE.exec(line);
    if (!m) continue;
    const [, ip, path, status, ua] = m;
    const bot = Object.keys(BOTS).find((n) => ua.toLowerCase().includes(n.toLowerCase()));
    if (!bot) continue;
    const s = (stats[bot] ||= { job: BOTS[bot], hits: 0, ok: 0, robots: 0, ips: new Set(), paths: {} });
    s.hits++;
    if (status[0] === "2") s.ok++;
    if (path === "/robots.txt") s.robots++;
    s.ips.add(ip);
    s.paths[path] = (s.paths[path] || 0) + 1;
  }
  const top = (paths) => Object.entries(paths).sort((a, b) => b[1] - a[1])[0][0];
  console.table(
    Object.entries(stats)
      .sort((a, b) => b[1].hits - a[1].hits)
      .map(([bot, s]) => ({ bot, job: s.job, hits: s.hits, "2xx %": Math.round((100 * s.ok) / s.hits), robots: s.robots, ips: s.ips.size, top: top(s.paths) }))
  );
}
main(process.argv[2]).catch((e) => { console.error(e.message); process.exit(1); });

On a host without raw access logs, middleware gives you the same signal. This is Next.js 13 Pages Router syntax; Next.js 16 renamed the convention to proxy.js, so adjust if you upgrade. It runs on every matched request, so keep it cheap: it is a measurement tool, not a firewall.

middleware.js — one JSON line per AI-bot request

// middleware.js  (project root, next to pages/)  ·  Next.js 13 Pages Router
import { NextResponse } from "next/server";

const AI_BOTS =
  /(OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|meta-webindexer|meta-externalagent|meta-externalfetcher|Google-Agent)/i;

export function middleware(request) {
  const ua = request.headers.get("user-agent") || "";
  const hit = ua.match(AI_BOTS);
  if (hit) {
    // one JSON line per hit: grep it, or ship it with your host's log drain
    const ip = request.headers.get("x-forwarded-for");
    console.log(JSON.stringify({ type: "ai-bot", bot: hit[1], path: request.nextUrl.pathname, ip, t: new Date().toISOString() }));
  }
  return NextResponse.next();
}

// run on pages and robots.txt, skip build assets
export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

7. Step 5 — Verify the Bot Is Real

A user agent string is a claim any script can make. Before a number goes into a report, check the source IP: OpenAI, Anthropic, Perplexity, Google and Apple publish IP ranges as JSON lists of CIDR prefixes, and Google and Apple document reverse DNS too (Google: reverse lookup, hostname under googlebot.com, google.com or googleusercontent.com, then forward lookup back to the same IP).

scripts/verify-bot-ip.js — published ranges plus reverse and forward DNS

// node scripts/verify-bot-ip.js 203.0.113.7 GPTBot   (Node 18+: global fetch)
const { BlockList, isIP } = require("node:net");
const dns = require("node:dns").promises;

// IP range files published by each vendor: { prefixes: [{ ipv4Prefix | ipv6Prefix }] }
const RANGES = {
  GPTBot: "https://openai.com/gptbot.json",
  "OAI-SearchBot": "https://openai.com/searchbot.json",
  "ChatGPT-User": "https://openai.com/chatgpt-user.json",
  ClaudeBot: "https://claude.com/crawling/bots.json", // one list for Anthropic's bots
  PerplexityBot: "https://www.perplexity.com/perplexitybot.json",
  Googlebot: "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
  Applebot: "https://search.developer.apple.com/applebot.json",
};
// Google and Apple also document reverse DNS (hostname suffixes)
const RDNS = {
  Googlebot: [".googlebot.com", ".google.com", ".googleusercontent.com"],
  Applebot: [".applebot.apple.com"],
};

async function inRanges(bot, ip) {
  const { prefixes } = await (await fetch(RANGES[bot])).json();
  const list = new BlockList();
  for (const p of prefixes) {
    const [addr, len] = (p.ipv4Prefix || p.ipv6Prefix).split("/");
    list.addSubnet(addr, Number(len), p.ipv4Prefix ? "ipv4" : "ipv6");
  }
  return list.check(ip, isIP(ip) === 6 ? "ipv6" : "ipv4");
}

// reverse DNS -> hostname ends with a documented suffix -> forward DNS returns the same IP
async function rdnsOk(bot, ip) {
  try {
    for (const host of await dns.reverse(ip)) {
      if (!RDNS[bot].some((s) => host.endsWith(s))) continue;
      if ((await dns.lookup(host, { all: true })).some((a) => a.address === ip)) return true;
    }
  } catch (e) {
    // no PTR record: not verified
  }
  return false;
}

(async () => {
  const [ip, bot] = process.argv.slice(2);
  if (!RANGES[bot]) throw new Error("unknown bot: " + bot);
  console.log({ bot, ip, inRanges: await inRanges(bot, ip), rdnsOk: RDNS[bot] ? await rdnsOk(bot, ip) : "n/a" });
})();

Cache the JSON files, they change. Anthropic warns that blocking by IP can stop its crawler from reading your robots.txt and may not give a reliable opt-out, so use ranges to verify what you already logged and robots.txt for policy. For vendors whose pages I read without an IP list (Meta, for one), treat user-agent-only counts as unverified.

8. Step 6 — Structured Data and the Vendors’ Own Reports

In the Pages Router, JSON-LD is a plain script tag with type application/ld+json inside next/head. The Next.js docs recommend a native script tag rather than next/script, and warn that JSON.stringify does not sanitize strings, so replace < with its unicode escape. Give the tag a key so it is not duplicated across renders.

components/JsonLd.js — Pages Router

// components/JsonLd.js  (Pages Router: render inside next/head)
import Head from "next/head";

export default function JsonLd({ id, data }) {
  return (
    <Head>
      <script
        key={id}
        type="application/ld+json"
        dangerouslySetInnerHTML={{
          // JSON.stringify does not sanitize: escape "<" so a value can never close the tag
          __html: JSON.stringify(data).replace(/</g, "\\u003c"),
        }}
      />
    </Head>
  );
}

// pages/blog/my-post.js
<JsonLd
  id="article-ld"
  data={{
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    headline: "Measuring AI search visibility",
    datePublished: "2026-10-14",
    dateModified: "2026-10-14",
    author: { "@type": "Person", name: "Diego Rodriguez", url: "https://www.diego-rodriguez.work" },
  }}
/>

9. A Measurement Routine That Holds Up

Measurement decays when it is a one-off. This is the cadence I would run on any site that cares about AI visibility, in the order the data gets cheaper to read:

10. Common Errors (All of Them Fixable in an Afternoon)

Almost every misleading AI-traffic number I have seen falls into one of these buckets. The left column is what trustworthy reports do; the right column is what produces a confident wrong slide.

Conclusion

You now have three instruments: a bot inventory taken from vendor docs, a robots.txt that encodes one policy per job, and two reports (GA4 by channel and engagement, logs by bot) that you can verify against published IP ranges. None of them tells you ’how visible am I in AI answers’ on its own. Together they tell you what you can and cannot claim.

This week: run both scripts once, write down three numbers (human sessions, AI Assistant sessions, verified AI-bot hits) and stop reporting the raw Direct line. Then use the GEO posts below to decide what to change, and come back in a month to see whether the numbers moved.

Sources

• OpenAI — Overview of OpenAI crawlers (OAI-SearchBot, GPTBot, ChatGPT-User)
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity — PerplexityBot and Perplexity-User
• Google — Common crawlers (Googlebot, Google-Extended)
• Google — User-triggered fetchers
• Google — Verifying requests from Google crawlers and fetchers
• Google Search — AI features and your website
• Google Search — Create and submit a robots.txt file
• Apple — About Applebot (Applebot-Extended, reverse DNS, IP list)
• Meta — Web crawlers (meta-webindexer, meta-externalagent, meta-externalfetcher)
• Google Analytics Help — Default channel group (AI Assistant)
• Google Analytics Help — Custom channel groups
• Google Analytics Help — About regular expressions
• Google Analytics Help — Known bot-traffic exclusion
• Google Analytics — Release notes (AI Assistant channel, May 13 2026)
• Next.js — How to implement JSON-LD
• Next.js — proxy.js (formerly middleware.js) and matcher
• Bing Webmaster Blog — Introducing AI Performance (public preview)
• Clickport — Why ChatGPT traffic shows as Direct in GA4 (vendor data)
• Search Engine Land — Microsoft Bing/Copilot use schema for its LLMs (secondary report)