Applebot User-Agent & Preparing for Apple Intelligence Search

So preparing for Apple Intelligence search is mostly crawler work. The crawler has to reach your server, be allowed by robots.txt, render the page, find nothing that tells it to stay quiet, and then rank. Each of those steps can fail on its own, and Apple's crawler page points to no crawl report or console for site owners, only an email address. You check each step yourself.

The timing matters. Apple put Siri AI into beta on 14 September 2026, and its launch note says it answers "questions from the web on virtually any topic" and sits inside Spotlight on iPad and Mac. Apple's About Applebot page carries a publish date of 4 September 2026, and it still did on 1 October. Many sites have already made a choice: Rankbox's AI bot crawler census found 44.63% of news and media sites, and 5.94% of all 21,353 sites, fully block Applebot-Extended.

This guide covers the crawler itself: what it powers, which switch does what, how it renders, the robots.txt rules Apple applies, server checks and Apple's ranking factors, ending in a five-gate readiness check. For the exact user-agent strings and how to prove a request is really Apple's, see our reference on Applebot user agent strings and verification. For how Siri decides between its own answer and ChatGPT, see our guide to Apple Intelligence routing.

What Applebot Powers in October 2026

Apple's support page lists three jobs for the crawl, each with its own off switch.

Search in Spotlight, Siri and Safari

This is the original job. Apple says the data it gathers powers "the search technology integrated into many user experiences" across its devices, naming Spotlight, Siri and Safari. Allowing Applebot in robots.txt lets your pages appear in those results "for Apple users around the world."

The archived App Search Programming Guide, last updated in 2016, explains where those pages live: the crawler stores them in "Apple's server-side index," which feeds Spotlight and Safari search results. That guide is no longer maintained, so treat it as background.

Context for AI answers

The second job is newer. Applebot's data may give Apple's models current context "when AI models are used to generate output" in Apple products. Apple's example is answering broad general-knowledge questions in Siri and Search, with links to the sites used. In June 2026 Apple said Siri AI on iPad and Mac is part of Spotlight, so on those devices a Spotlight search can turn into an AI answer.

Training data for Apple's models

The third job is training. Apple's training data page, dated 9 September 2026, says Apple trains its generative models partly on "publicly available information crawled by Apple's web crawler Applebot," after filtering and plain-text extraction. Apple's third-generation model announcement of 8 June 2026 adds that Apple respects "the rights of web publishers to opt out." That opt-out is Applebot-Extended.

What Applebot doesn't power

Two Apple features sound like crawler work but aren't, going by Apple's own descriptions:

• Safari page summaries. Apple's iOS 26 guide says Apple Intelligence can summarize the webpage you have open, and in iOS 27, Reader adds "a summary and table of contents" to longer pages. Both work from the page in front of the reader. Apple doesn't connect either to its crawler.
• Podcast fetches. Apple's page names a separate agent, iTMS, that may come from the same applebot.apple.com hosts. It only fetches URLs tied to content registered on Apple Podcasts, and it doesn't follow robots.txt because it isn't a general search crawler.

Applebot vs Applebot-Extended: Which Switch Does What

Apple gives publishers several controls, and they overlap in confusing ways. The table below sets each one against the four uses of the crawl, in Apple's own wording, checked on 1 October 2026.

Applebot-Extended is a rule, not a visitor

Apple calls Applebot-Extended a "secondary user agent," but it also says plainly that it "does not crawl webpages." It's only used to decide what Apple may do with data the main crawler already fetched. That has two practical effects. You'll never see it in your logs, so no firewall rule can match it. And disallowing it costs you nothing in search: Apple says those rules "are not considered in ranking for Search."

Apple's own robots.txt example disallows only /private/ for the token, which shows you can opt out section by section. Most sites that opt out use Disallow: / for the whole site.

The page-level tags Applebot honors

Apple supports the standard robots meta tag and an Apple-only version, plus the HTTP header. Here's what each directive means to the crawler, in Apple's terms:

Three details are easy to miss. First, <meta name="applebot" content="nosnippet"> targets Apple alone, leaving Google's snippets untouched. Second, the header form X-Robots-Tag: applebot: nosnippet is how you tag PDFs and images, which can't carry meta tags, and it never shows up in a page's source. Third, the paywall flag works only on whole pages: Apple says "Section-level markup using hasPart is not supported." Our glossary entry on snippet controls explains how these tags work across search engines.

How Applebot Renders Pages

Apple says outright that its crawler renders. The support page puts it this way: "Applebot may render the content of your website within a browser."

What Apple says it does

The support page is short. If robots.txt blocks the JavaScript, CSS, XHR requests or other files a page needs, Applebot "may not be able to render the content properly." Apple gives you two ways out: let it fetch everything a person's browser would, or make sure the page still makes sense when some files are missing, which Apple calls "graceful degradation."

Apple's machine learning team went further in a June 2025 report. For training data, it described "headless rendering, enabling full-page loading, dynamic content interaction, and JavaScript execution," plus "interaction simulation" for pages that depend on clicks. That describes the training pipeline, not search, so don't read it as proof that every search fetch runs every script. The support page's "may render" is the safer promise.

That sets Apple apart. In Vercel and MERJ's December 2024 study, none of the major AI crawlers they measured ran JavaScript, including OpenAI's, Anthropic's and Perplexity's. A client-rendered page can therefore look fine to Apple and blank to them. Our guide to dynamic rendering and prerendering for AI crawlers covers the fix per framework.

Four things that hide your content from Applebot

1. Blocked files. A Disallow that covers /assets/, /static/ or /api/ can strip the scripts and data a page needs. Check this first, because it often comes from the Googlebot fallback described in the next section.
2. Logins and paywalls. Apple's crawler privacy page, published 9 September 2026, says Applebot "does not crawl data from websites that require login credentials or that are protected by a paywall." Its main crawler page, meanwhile, says pages flagged isAccessibleForFree: false can still appear in search. Apple doesn't explain how the two fit, so keep anything you want found outside the wall.
3. Hash routes. The archived 2016 guide says "Applebot ignores the fragment identifier component of a URL." If your pricing tabs live at /pricing#agency and /pricing#enterprise, Apple may see one URL. Give each view its own path.
4. Text in images. If your prices only appear inside a graphic, a renderer that reads text won't find them. Put key facts in HTML text.

A quick render test

Open a key page in Chrome, then open DevTools, go to the Network panel and block the URL patterns your robots.txt disallows for Applebot (or for Googlebot, if Applebot isn't named). Reload. What's left on screen is roughly what Apple can render. For the stricter case, what a crawler that runs no JavaScript gets, view the page source, or run the free AI search readiness check, which fetches raw HTML only. See our glossary on server-side rendering for why the raw HTML matters to other crawlers.

The robots.txt Rules Apple Applies

Apple says Applebot "respects standard robots.txt directives" in its search crawls. The standard is RFC 9309, and Apple follows it with one big exception.

The fallback order

1. A group that names Applebot. If one exists, that's the group used.
2. Your Googlebot group. If robots.txt never mentions Applebot but does mention Googlebot, Apple says "the Apple robot will follow Googlebot instructions." The page still said this on 1 October 2026.
3. The * group. If neither is named, the standard rule applies, and the crawler obeys the group for all user agents.

Step 2 is not part of RFC 9309, which sends unnamed crawlers straight to *. It means a rule you wrote for Google years ago can quietly govern Apple today. Apple doesn't say whether a group for a Google variant, such as Googlebot-Image, counts, so naming Applebot removes the guesswork. For every other bot's tokens, see our AI crawler directory. For which AI bots to block by goal, with ready-made files, see our AI crawler robots.txt guide, and for the basic syntax, our robots.txt glossary entry.

Crawl-delay doesn't work

"Applebot does not follow crawl-delay," Apple says. Instead, its "crawl rate adjusts automatically when a site slows down or returns errors," and Apple caches what it fetched to avoid repeat visits. If you need to throttle it hard, do it at the server with rate limits, after confirming the requests are genuine.

A tested template

This file keeps Applebot in search, opts out of training and stops the Googlebot fallback from deciding anything. Adjust the private paths to your own site.

# Apple: opt out of model training only
User-agent: Applebot-Extended
Disallow: /
 
# Apple: stay in Spotlight, Siri and Safari search
User-agent: Applebot
Disallow: /account/
 
# Google keeps its own rules
User-agent: Googlebot
Disallow: /account/
 
User-agent: GPTBot
Disallow: /
 
User-agent: *
Disallow: /account/

On 1 October 2026 this file was run through three parsers: Python 3.14.3's built-in urllib.robotparser, Protego 0.7.0 and the npm package robots-parser 3.0.1. All three agreed: Apple's crawler may fetch /, /pricing and /assets/app.js but not /account/settings, and the training token is blocked everywhere. Two layout choices keep simpler parsers honest. Python's parser takes the first group whose name appears inside the bot's name, so the longer Applebot-Extended group goes first. It also applies rules in file order, so the template leaves out an Allow: / line that would otherwise win before the Disallow.

Why testers miss the fallback

The same three parsers were run on a file with no Apple group and a Googlebot group that disallows /assets/. All three said Apple's crawler could fetch /assets/app.js, because they fall back to *. Apple's rule says the opposite. So when Apple isn't named, test your paths as Googlebot. Better still, name it.

Server Checks Before Apple Intelligence Search Reaches You

robots.txt only matters if the crawler's request gets an answer. Four server-side checks catch most of the rest.

Status codes

Apple doesn't publish how its crawler treats each HTTP status code. It only says the crawler slows down when your site returns errors. For robots.txt itself, RFC 9309 sets the baseline: if the file returns a 4xx status, a crawler "MAY access any resources," and if it returns a 5xx, the crawler "MUST assume complete disallow." A robots.txt that errors during deploys can pause Apple's crawl. Serve it as a static file with a 200.

For pages, use plain HTTP: 200 for live pages, 301 for moved ones, 404 or 410 for pages that are gone. Avoid soft errors, where a missing page returns 200 with a "not found" message.

Speed

Apple gives no speed threshold. What it does say is that the crawler backs off when your site slows down, so a slow site gets crawled less. "Webpage design characteristics" is also one of Apple's ranking factors, though Apple doesn't define it.

Firewalls and CDNs

A block can also sit outside robots.txt. Cloudflare's July 2026 announcement says that from 15 September, "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," whether through its new AI traffic options or the older Block AI bots toggle. Its changelog says the same. Our Cloudflare challenge trap guide shows where to see the block and how to block trainers by name instead, and our guide to Cloudflare AI bot management walks through each setting.

Before you let anything through a firewall because it claims Apple's name, confirm it's real. Apple publishes a reverse DNS rule and an IP list, and they don't fully overlap. Our Applebot user agent reference shows the tested method.

Recognize Applebot in your logs

Apple documents one general format and two example strings, for a Mac and an iPhone. Both end in (Applebot/0.1; +http://www.apple.com/go/applebot), and Apple warns that the browser version in front of that may change. So match the Applebot/ token, not the whole string. Requests from iTMS come from the same hosts but serve Apple Podcasts. Our Applebot user agent reference lists the exact strings, log one-liners and a verification script run against Apple's own list.

A three-command curl check

These commands show what your server returns to a request carrying Apple's desktop crawler string. Replace the URL with one of yours.

URL="https://www.example.com/pricing"
UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)'
curl -s -o /dev/null --max-time 20 -A "$UA" -w 'page %{http_code} %{time_total}s\n' "$URL"
curl -s -D - -o /dev/null --max-time 20 -A "$UA" "$URL" | grep -i '^x-robots-tag' || echo 'no X-Robots-Tag header'
curl -s --max-time 20 -A "$UA" "$URL" | grep -ioE '<meta[^>]+name="?(robots|applebot)"?[^>]*>' || echo 'no robots or applebot meta tag'

These commands were run on 1 October 2026 against a local test server that sent X-Robots-Tag: applebot: nosnippet and an Apple-only meta tag, and they printed both. Against example.com they printed a 200 and no tags.

One caution: this only tests your origin's response to the name. If your CDN checks crawler IPs, it may refuse your laptop's imitation while letting the real crawler through, or the reverse. Your access logs show what Apple actually received.

Apple's Ranking Factors and What You Control

Apple's crawler page lists five factors its search may use, and adds that they carry no set weights. The table turns each into something you can act on. The last column says how firm the link is.

Two notes. First, Apple's Search and Privacy statement says Safari and Spotlight send Apple limited data, including location and the suggestions people choose, without linking it to their account. That's the likely source of the engagement and location signals, though Apple doesn't say so directly. Second, these factors describe Apple's web search. Apple doesn't say whether Siri AI picks sources for its answers the same way.

An old Apple note hints at how engagement works. Its archived 2015 App Search FAQ told app makers that their "engagement-to-shown ratio" would be "an important factor." That was about app results a decade ago, so it's history, not a current rule.

For the routing side of Siri, including when ChatGPT answers instead, our Apple Intelligence routing guide has a three-door audit. For every assistant's crawler and listing at once, use the voice search optimization checklist.

The Five-Gate Applebot Readiness Check

A page reaches an Apple answer only if it clears five gates in order. A failure early on makes everything after it irrelevant, so audit top-down. We call this the Five-Gate Applebot Readiness Check: Reach, Permit, Render, Use and Rank, with three checks each.

Score one point per pass. Skip checks that can't apply, and divide by what's left.

Worked example: Tallyfold

Tallyfold is a fictional invoicing and payments app for agencies, with fictional rivals Brindlework and Kestrelyn, at the reserved domain tallyfold.example. Everything here is invented to show the method. Its pricing page should let Siri say one thing correctly: Tallyfold costs $39 a month with 3 users included, then $12 per extra user, so a five-person agency pays $39 + 2 × $12 = $63 a month.

Tallyfold's marketing site is a single-page React app. Its first audit:

That's 7 of 15, or 47%. Notice that the Rank gate passes while the page still can't appear. Good content was never the problem.

The fixes, in order of reach and effort:

1. Cloudflare (R1). Set the Training preset back to Allow and block training crawlers by name, so the multi-purpose rule stops catching Apple's crawler.
2. robots.txt (P1, P2, P3). Swap in the template above. Apple's crawler gets its own group, the asset and API paths open, and the training token carries the training opt-out that Cloudflare used to.
3. Rendering follows (D1). With scripts and the plans API now allowed, Apple can render the prices. No code change needed.
4. Headers and markup (U1, U3). Remove the noindex header from public PDFs, and set the paywall flag only on the two paid reports that are really gated.

Those four steps fix seven checks, so Tallyfold moves from 7 to 14 of 15, or 93%. The last fail, D2, needs a developer: each plan gets its own path, such as /pricing/agency. That brings it to 15 of 15.

One more step serves crawlers beyond Apple. Since most AI crawlers run no JavaScript, Tallyfold should also server-render the pricing page, so the $63 example is in the HTML from the first byte. Apple would read the page either way. Others won't.

None of this guarantees that Siri will mention Tallyfold. It removes every documented reason it can't.

Where Rankbox Fits

Rankbox works on the Rank gate, mostly check K1. Answer-Space Research maps the questions buyers ask AI assistants and search engines, with volume, difficulty and intent shown as model estimates. The Citation-Ready Writer then researches the live web and drafts source-backed articles of 2,000 to 3,500 words that open with a plain answer, and articles reach your site through Rankbox's API.

The first four gates are yours or your host's. Rankbox doesn't run a CDN, manage robots.txt, change headers or prerender pages, and it doesn't track whether Siri or any other assistant cites you. The Business plan is $49.50 a month, charged 7 days after you start the free trial unless you cancel first, and the trial takes a card; see pricing.