Edge SEO for AI: Dynamic Rendering & Header Injection via Cloudflare Workers

Edge SEO is older than the AI use. Dan Taylor, who says he was credited with coining the term at TechSEO Boost in 2018, defines it as shipping technical SEO fixes "through a serverless application, such as Cloudflare Workers, deployed on a CDN." What changed in 2026 is who fetches the page. Agent tools now ask for Markdown by name, and the savings are large: Roots measured one WordPress.org docs page at 271,268 bytes as HTML and 44,841 bytes as Markdown.

This guide covers edge SEO for AI end to end: Google's cloaking line, what AI vendors document, a Cloudflare Worker printed in full and run locally, the headers to inject, when Cloudflare's built-in feature replaces the code, and how to test and roll back. New to the topic? Start with our plain definition of edge SEO or the practical guide to SEO changes at the CDN.

What Edge SEO for AI Changes at the CDN

A CDN worker sits between the visitor and your origin server. It reads the request, fetches the page, and can change the response before anyone sees it. Classic edge SEO uses that spot for five jobs, and Google documents a form of each.

Our practical edge SEO guide walks through those jobs platform by platform. The rest of this post is the AI layer on top.

Why edge SEO for AI is worth the effort

1. Many AI crawlers read raw HTML. In December 2024, Vercel and MERJ reported that "none of the major AI crawlers currently render JavaScript," naming bots from OpenAI, Anthropic, Meta, ByteDance and Perplexity. Googlebot does render. Our guide to dynamic rendering and prerendering for AI crawlers covers client-rendered pages.
2. Agents pay per token. Menus, scripts and footers fill an agent's context window and answer nothing.
3. Agents can say what they want. Some agent tools send an Accept header that lists Markdown first, which turns the job into ordinary HTTP content negotiation.

Edge SEO for AI Without Cloaking: Where Google Draws the Line

Anything that serves bots a different response from people sits next to Google's cloaking rule. Start with Google's own words.

What Google's pages say, word for word

Google's spam policies (updated 28 August 2026) define cloaking as "presenting different content to users and search engines with the intent to manipulate search rankings and mislead users." One example is "inserting text or keywords into a page only when the user agent that is requesting the page is a search engine, not a human visitor."

Google's page on dynamic rendering (updated 10 December 2025) covers the format case: "Googlebot generally doesn't consider dynamic rendering as cloaking. As long as your dynamic rendering produces similar content, Googlebot won't view dynamic rendering as cloaking." Serving "completely different content to users and crawlers can be considered cloaking," such as "a page about cats to users and a page about dogs to crawlers." The same page calls dynamic rendering "a workaround and not a long-term solution." Serving Markdown on request is a close cousin: one URL, a second rendering for a different client. Our post on dynamic rendering SEO covers Google's history with the technique.

So the line is plain. The same facts and content in a different format is acceptable. Different content for bots is cloaking. A Markdown copy that drops your menu is a format change. A Markdown copy that adds claims, keywords or prices the HTML doesn't show is cloaking, whoever receives it.

Google's rules govern Google Search, and the crawler pages from OpenAI, Anthropic and Perplexity set out no cloaking rule of their own. The same test still protects you: an answer engine that quotes your Markdown should quote what a buyer reads on the page.

Two more points from Google matter. Its AI optimization guide says you don't need "Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." And when a developer served Markdown to GPTBot and ClaudeBot by user agent, John Mueller pushed back in February 2026: "it seems very different to serve it a text file when they're looking for a HTML page."

What AI vendors document about their requests

Before you build edge SEO for AI around a bot, check what its vendor says it sends.

All rows checked on 1 October 2026. Our AI crawler directory has the full user-agent list.

Why content negotiation beats user-agent sniffing

Matching user agents to decide who gets Markdown has three flaws. Crawler vendors don't say their bots want Markdown, so you'd be guessing. Every new bot needs a new rule. And Vary: User-Agent splits your cache into a copy per browser version. Content negotiation flips it around: the client states the formats it accepts, weighted by q values under RFC 9110, and the server answers. A crawler that asks for HTML gets HTML, which settles Mueller's objection. So this edge SEO for AI pattern starts from the header, and user-agent matching stays an off-by-default fallback for a client whose vendor documents that it reads Markdown without the header.

Cloudflare Workers and HTMLRewriter: The Edge SEO for AI Toolkit

A Worker is a JavaScript function that Cloudflare runs on a route you choose, such as tallyfold.example/pricing*. It receives each request, calls fetch() to reach your cache or origin, and returns a response. Cloudflare says Workers run before the cache, so a Worker can shape every response without the origin knowing.

HTMLRewriter is the Workers API for editing HTML as it streams. You attach handlers to CSS-style selectors with .on(). An element handler can read and set attributes, insert content around a tag, or call onEndTag() when the element closes. A text handler receives the text inside it. Cloudflare's HTMLRewriter docs flag two details that bite:

• Text arrives in chunks. One text node "can be represented by multiple chunks," so join them before matching on text.
• A thrown error stops the parse. If a handler throws, "parsing is immediately halted," and a response already streaming arrives cut short.

A third detail showed up when the Worker below ran: text chunks keep their HTML entities, so & stays & until you decode it.

As of 1 October 2026, Workers pricing gives the Free plan 100,000 requests a day with 10 ms of CPU time per request. The Paid plan starts at $5 a month with 10 million requests and 30 million CPU milliseconds included, then $0.30 per extra million requests. Watch the 10 ms Free limit: converting a big page costs far more CPU than adding a header.

Cloudflare Workers Explained

Edge SEO for AI in Practice: A Markdown Worker

The Worker below serves the same page as Markdown when a client asks for it. It does four things:

1. Reads the Accept header. Markdown goes out only when the client names text/markdown and weights it at least as high as text/html.
2. Fetches the normal HTML page from the cache or origin, so the cache only ever holds HTML.
3. Converts the main content with HTMLRewriter: headings, paragraphs, list items and links, minus scripts, menus and forms.
4. Adds headers: Vary: Accept on both formats, a Link to the Markdown copy on HTML pages, and a canonical Link on the Markdown.

It also answers /pricing.md style URLs, which llms.txt files often point to. Skip that branch if your origin already serves real .md files. Set ROOT to the element that wraps your content.

// Same page, cleaner format: serve Markdown when a client asks for it.
const ROOT = "main"; // the element that wraps your page content
const BLOCKS = { h1: "# ", h2: "## ", h3: "### ", p: "", li: "- " };
const SKIP = ["script", "style", "nav", "noscript", "svg", "form"];
const ENTITIES = { amp: "&", lt: "<", gt: ">", quot: '"', nbsp: " " };
const isHtml = (res) => (res.headers.get("Content-Type") || "").includes("text/html");
const safeUrl = (href, base) => {
  try {
    return new URL(href, base).href;
  } catch {
    return null;
  }
};
// HTMLRewriter passes raw text, so entities such as &amp; arrive encoded.
const decode = (s) =>
  s.replace(/&(#\d+|[a-z]+);/gi, (m, e) =>
    e[0] === "#" ? String.fromCodePoint(Number(e.slice(1))) : (ENTITIES[e.toLowerCase()] ?? m),
  );
 
// The weight (q-value, RFC 9110) a client gives a media type; 0 = not acceptable.
function quality(accept, type, useWildcards) {
  const weights = {};
  for (const part of accept.toLowerCase().split(",")) {
    const [range, ...params] = part.split(";").map((s) => s.trim());
    const q = params.find((p) => p.startsWith("q="));
    weights[range] = q ? Number(q.slice(2)) || 0 : 1;
  }
  if (!useWildcards) return weights[type] ?? 0;
  return weights[type] ?? weights[type.split("/")[0] + "/*"] ?? weights["*/*"] ?? 0;
}
 
function wantsMarkdown(request, env) {
  const accept = request.headers.get("Accept") || "";
  const md = quality(accept, "text/markdown", false); // must be named, not implied by */*
  if (md > 0) return md >= quality(accept, "text/html", true); // a tie goes to Markdown
  // Fallback, off by default: tokens for clients whose vendors document
  // that they read Markdown but don't send the Accept header.
  const tokens = (env.MARKDOWN_UA || "")
    .split(",")
    .map((t) => t.trim())
    .filter(Boolean);
  return tokens.some((t) => (request.headers.get("User-Agent") || "").includes(t));
}
 
async function toMarkdown(html, pageUrl) {
  const blocks = [];
  const meta = { title: "", canonical: pageUrl };
  let current = null;
  let skipDepth = 0;
  const open = (prefix) => {
    current = { prefix, text: "" };
    blocks.push(current);
  };
  const add = (s) => {
    if (!current) open("");
    current.text += s;
  };
  const rw = new HTMLRewriter()
    .on("head > title", {
      text(t) {
        meta.title += t.text;
      },
    })
    .on('link[rel="canonical"]', {
      element(el) {
        meta.canonical = safeUrl(el.getAttribute("href"), pageUrl) || pageUrl;
      },
    })
    .on(ROOT, {
      text(t) {
        if (!skipDepth) add(t.text);
      },
    })
    .on(`${ROOT} a[href]`, {
      element(el) {
        const href = safeUrl(el.getAttribute("href"), meta.canonical);
        if (skipDepth || !href) return; // a broken link stays as plain text
        add("[");
        el.onEndTag(() => add(`](${href})`));
      },
    });
  for (const [tag, prefix] of Object.entries(BLOCKS)) {
    rw.on(`${ROOT} ${tag}`, {
      element(el) {
        if (skipDepth) return;
        open(prefix);
        el.onEndTag(() => {
          current = null;
        });
      },
    });
  }
  for (const tag of SKIP) {
    rw.on(`${ROOT} ${tag}`, {
      element(el) {
        skipDepth++;
        el.onEndTag(() => {
          skipDepth--;
        });
      },
    });
  }
  await rw
    .transform(new Response(html, { headers: { "Content-Type": "text/html" } }))
    .arrayBuffer();
  let body = "";
  let prev = null;
  for (const b of blocks) {
    const text = decode(b.text)
      .replace(/\s+/g, " ")
      .replace(/\[ /g, "[")
      .replace(/ \]/g, "]")
      .trim();
    if (!text) continue;
    if (body) body += b.prefix === "- " && prev === "- " ? "\n" : "\n\n";
    body += b.prefix + text;
    prev = b.prefix;
  }
  const front = `---\ntitle: ${JSON.stringify(decode(meta.title.trim()))}\ncanonical: ${meta.canonical}\n---\n\n`;
  return { markdown: front + body + "\n", canonical: meta.canonical };
}
 
export default {
  async fetch(request, env, ctx) {
    ctx.passThroughOnException(); // in production, an uncaught error falls through to the origin
    if (!["GET", "HEAD"].includes(request.method)) return fetch(request);
    const url = new URL(request.url);
    const isMdPath = url.pathname.endsWith(".md");
    const vary = env.MARKDOWN_UA ? "Accept, User-Agent" : "Accept";
 
    if (!isMdPath && !wantsMarkdown(request, env)) {
      const res = await fetch(request); // people and search engines: the HTML, untouched
      if (!isHtml(res)) return res;
      const out = new Response(res.body, res);
      const mdUrl = url.pathname === "/" ? "/index.md" : url.pathname.replace(/\/$/, "") + ".md";
      out.headers.append("Vary", vary);
      out.headers.append("Link", `<${mdUrl}>; rel="alternate"; type="text/markdown"`);
      return out;
    }
 
    // Fetch the normal HTML page, so the cache only ever stores HTML.
    const htmlUrl = new URL(url);
    if (isMdPath) htmlUrl.pathname = url.pathname === "/index.md" ? "/" : url.pathname.slice(0, -3);
    const headers = new Headers(request.headers);
    headers.set("Accept", "text/html");
    const res = await fetch(new Request(htmlUrl, { method: "GET", headers }));
    if (!res.ok || !isHtml(res)) return res;
    const html = await res.text();
    try {
      const { markdown, canonical } = await toMarkdown(html, htmlUrl.href);
      return new Response(request.method === "HEAD" ? null : markdown, {
        headers: {
          "Content-Type": "text/markdown; charset=utf-8",
          "Cache-Control": res.headers.get("Cache-Control") || "public, max-age=300",
          Vary: vary,
          Link: `<${canonical}>; rel="canonical"`,
        },
      });
    } catch (err) {
      console.error("markdown conversion failed", htmlUrl.pathname, err);
      const fallback = new Response(request.method === "HEAD" ? null : html, res);
      fallback.headers.append("Vary", vary);
      return fallback; // same page, original format
    }
  },
};

The wrangler.toml scopes the Worker to HTML paths, so images and scripts skip it:

name = "tallyfold-edge-markdown"
main = "src/index.js"
compatibility_date = "2026-09-30"
routes = [
  { pattern = "tallyfold.example/pricing*", zone_name = "tallyfold.example" },
  { pattern = "tallyfold.example/docs/*", zone_name = "tallyfold.example" }
]

How this Worker was tested

It ran under wrangler 4.146.0 in local mode (wrangler dev --local --local-upstream 127.0.0.1:8799) in front of a Python static server holding a fictional Tallyfold pricing page. wrangler deploy --dry-run also built it with the routes above. Nothing was deployed to a live zone.

The sample page shrank from 2,185 bytes to 840. Untested: a live zone, the Free plan's 10 ms CPU limit, and very large pages. Locally, an uncaught error returned a 500 rather than reaching the origin, so the code catches conversion errors itself.

The converter is minimal on purpose. It flattens tables, ignores code blocks and decodes five named entities plus numeric ones. Table-heavy pages need Cloudflare's converter or build-time Markdown.

Header Injection for AI Crawlers and Agents

Headers are the cheapest edge change, and most edge SEO for AI work is header work. Five earn their place:

Two cautions. Don't put noindex on a .md copy that already carries a canonical: Google doesn't recommend "using noindex to prevent selection of a canonical page within a single site." And skip Vary: User-Agent unless you really match user agents: Cloudflare's Workers Cache stores variants "per exact request-header value."

On Content-Signal, Cloudflare's converter adds ai-train=yes, search=yes, ai-input=yes when your origin sends no value, and keeps yours when you do. If you'd rather not offer pages for training, set the header at the origin before switching conversion on.

When Cloudflare's Markdown for Agents Replaces the Code

Cloudflare ships the same idea as a switch, so on Pro and above, edge SEO for AI can be one toggle. Markdown for Agents converts HTML to Markdown whenever a request lists text/markdown in its Accept header. Per its docs (updated 13 July 2026), responses carry text/markdown, Vary: Accept, token-count headers and YAML front matter, with navigation and scripts stripped and JSON-LD kept. It launched as a beta in February 2026; the current docs no longer use that word. It's available "to Pro, Business and Enterprise plans, and SSL for SaaS customers at no cost," under AI Crawl Control or per path with a Configuration Rule.

Cloudflare's converter does better on tables and structured data. The Worker's advantage is control over selectors, URLs and headers. Don't run both on one path, or you'll debug two converters at once.

Testing Edge SEO for AI, Rolling It Back, and the Risks

The Same-Page Parity Test

Any edge SEO for AI variant must carry the same facts as the page people see. The Same-Page Parity Test is six checks, run on every template before launch and after each template change:

Check 4 catches the real problems. This version compares dollar amounts:

URL=https://tallyfold.example/pricing
prices() { grep -oE '\$[0-9]([0-9,.]*[0-9])?' | sort -u; }
diff <(curl -s -H "Accept: text/html" "$URL" | prices) \
     <(curl -s -H "Accept: text/markdown" "$URL" | prices) && echo "prices match"

Against the local Worker it printed "prices match". Then a yearly price was placed in an <aside> outside <main>. The diff printed < $390 and no match line, because the converter only reads ROOT. That's drift in miniature: a price people see that agents never get.

Cache variance and Vary

Cloudflare's docs say that "by default, Cloudflare does not consider vary values in caching decisions," apart from Accept-Encoding, image variants and the Cache Rules Vary setting, which every plan has. If your origin did the negotiation itself, a cached Markdown copy could reach a browser. The safe edge SEO for AI pattern converts after the cache lookup, so the cache only ever holds HTML. Switch on Workers Cache in front of the Worker and each distinct Accept string becomes its own cached copy.

Rollback and failure modes

• Roll back in one step. wrangler rollback or the Deployments tab restores one of the last 100 versions at once, per Cloudflare's rollback docs.
• Fail open, within limits. ctx.passThroughOnException() sends a request to the origin when code throws. Cloudflare's context docs say it "does not mitigate failures such as exceeding CPU or memory limits."
• Debug with headers. curl -sI -H "Accept: text/markdown" shows which branch ran, and wrangler tail streams live logs.
• Check the bot settings. Cloudflare's AI bot controls run at the same layer and can block the agents this Worker serves. Our Cloudflare AI bot management walkthrough covers each setting.

The bigger risk isn't technical. Edge code is a second place where a page can change, so the Markdown and the CMS can drift apart. Give one team ownership, keep the Worker in version control, and rerun the parity test whenever a template changes.

Worked Example: Tallyfold's Edge SEO for AI Rollout

Tallyfold is a fictional invoicing and payments app for agencies, made up for this example. Its marketing site runs on an older CMS behind Cloudflare's Free plan. Its pricing page says $39 a month with 3 users included, then $12 for each extra user. Here is how its edge SEO for AI rollout runs.

The problem. In this example, Tallyfold's logs show agent requests for /pricing and /docs/ with Accept: text/markdown, text/html, */*, and its llms.txt links to /pricing.md, which returns a 404. The pricing HTML also carries a menu, scripts and a pricing-experiment flag that no agent needs.

The plan, in order.

1. No-code fixes first. Old 2024 plan URLs move to /pricing with Bulk Redirects, and staging gets X-Robots-Tag: noindex from a Transform Rule.
2. The Worker goes on /pricing* and /docs/* only.
3. The parity test runs on both templates. On the pricing page, status (200), canonical, the four headings, the three content links and all four prices ($12, $39, $84 and $123) match.
4. The budget check. Those paths get about 600,000 requests a month: 600,000 ÷ 30 = 20,000 a day, a fifth of the Workers Free plan's 100,000.

The numbers. The trimmed pricing page drops from 2,185 bytes of HTML to 840 bytes of Markdown, a saving of 1,345 bytes, or 61.6%. Both formats show the same price for a 10-person team: $39 + 7 × $12 = $123 a month.

What Tallyfold doesn't do. It doesn't show bots a lower price, add keyword text to the Markdown, or noindex the .md copies. If it moves to Cloudflare Pro ($20 a month billed annually, or $25 monthly, per Cloudflare's plans page on 1 October 2026), its edge SEO for AI setup shrinks: Markdown for Agents covers /docs/, and the Worker stays only for the .md URLs.

Where Rankbox Fits in Edge SEO for AI

Rankbox doesn't run a CDN, edge workers, prerendering or bot management, and it won't touch your headers or robots.txt. Its part comes earlier: Answer-Space Research maps the questions buyers ask AI engines, and the Citation-Ready Writer writes source-backed articles that reach your site through the Rankbox API. A clean Markdown copy only helps if the page underneath says something worth quoting.

Rankbox doesn't track AI citations today. To see what a non-rendering crawler gets from a page, try the free AI search readiness check. The Business plan is $49.50 a month, and the last step of setup is a 7-day free trial that takes a card; see pricing.