On this page11 sections
The short answer
To get cited by Meta AI, allow Meta-WebIndexer in robots.txt — Meta says doing so “helps us cite and link to your content in Meta AI’s responses” — and keep Meta-ExternalFetcher unblocked at your CDN. Serve the answer in server-rendered HTML, since Meta documents no JavaScript rendering. Then cover both places Meta AI looks: the open web, reached through unnamed search partners and Meta’s own index, and public posts on Facebook, Instagram and Threads, which it searches directly.
Key takeaways
Meta-WebIndexeris Meta AI’s search crawler: allowing it “helps us cite and link to your content,” per Meta.Meta-ExternalAgentcrawls for training.- Meta no longer names its search partners. It announced Bing in 2023 and Google in 2024; its May 2026 terms say only “select partners, like search engines.”
- Meta AI also runs a semantic search over Facebook, Instagram and Threads posts from January 2025 on. Your social presence is part of your index.
- Licensed publishers get linked news answers: Reuters since October 2024, and CNN, Fox News, USA Today, Le Monde and others since December 2025.
Meta-WebIndexerwent from about 2.2% to 37.8% of AI crawler requests in Promptwatch’s logs between mid-July and August 9, 2026.- GA4’s AI Assistant channel doesn’t name Meta AI, and Meta doesn’t document the referrer its links carry. Measure with a custom channel, logs and an API panel.
How Meta AI answers with the web#
Meta AI answers from the model when it can and searches when a question needs something current or specific. Meta’s Search grounding documentation, written for developers using the same model family, says the model “skips the search when it can answer confidently from its training data” and that simple factual questions “typically do not trigger a search.” Since April 8, 2026 that model is Muse Spark, the first release from Meta Superintelligence Labs. It launched in the Meta AI app and on meta.ai, and on May 12, 2026 Meta said it was starting to bring it to Meta AI in WhatsApp, Instagram, Facebook, Messenger and Threads.
What it searches is less documented than on any other major assistant. Meta announced Bing in 2023 and Google in 2024, but its current terms name no engine. Alongside those partners sit Meta’s own crawler, Meta-WebIndexer, and a separate search over posts on Meta’s apps. The tool names below come from Meta’s API docs and from a tool list Meta AI returned to Simon Willison on launch day, which a Meta engineer confirmed belongs to the new harness.
- 1
Meta AI decides whether to search
News, sports, prices, local and product questions search; stable knowledge comes from the model. Web search is a setting users can turn off; in incognito chats it “begins toggled on.”
Your lever: Decide on
Meta-ExternalAgentdeliberately: it collects training data, and unsearched answers come from what the model learned. - 2
It writes a main query plus alternatives
browser.searchtakes aprimary_query, a list ofalternative_queries, an optionalsincedate, and verticals for news, sports, weather, finance and local. For incognito chats, Meta’s help center says the query goes to “select search engines without connecting it to your account.”Your lever: Title pages the way people ask — product names, places, and the year where freshness matters.
- 3
Partners and Meta's own index return results
Willison’s tool dump describes the engine as undisclosed.
Meta-WebIndexerexists “to improve Meta AI search result quality,” and Meta ties it directly to citations and links.Your lever: Allow
Meta-WebIndexer, and keep ranking in Google and Bing, the partners Meta has used before. - 4
Meta's own posts are searched too
meta_1p.content_searchruns semantic search over Facebook, Instagram and Threads posts from 2025 onward, and shopping questions can query Meta’s product catalog and Marketplace.Your lever: Post on your own accounts and earn public posts from customers; both are retrievable.
- 5
It opens and reads the pages it needs
browser.openloads a full result andbrowser.findpattern-matches inside it. Meta doesn’t document which user agent does the reading or whether it runs JavaScript.Your lever: Put the answer in the raw HTML, near the top, and don’t block
Meta-ExternalFetcherat your WAF. - 6
The answer cites a subset of what it read
Links appear in the response, with a Sources list under it. In the API, the retrieved results are “every source the model considered” and the citations “the subset it actually cited.”
Your lever: State each fact in one self-contained sentence a link can attach to.
1B+
monthly actives already use Meta AI, per Mark Zuckerberg — the latest MAU figure in Meta's earnings calls
+60%
more people interacting with Meta AI each day since Meta rebuilt it on Muse Spark
37.8%
of AI crawler requests in Promptwatch's logs came from Meta-WebIndexer on Aug 9, 2026, up from ~2.2% in mid-July
Meta's five crawlers and robots.txt#
Meta documents five crawlers, each identified by a lowercase token such as meta-webindexer/1.1. Meta says it prefers “industry-standard practices like robots.txt rather than non-standard formats like NoAI tags,” and that its crawlers may cache robots.txt “for up to 24 hours.” There’s no FacebookBot on the current page, so an old rule targeting it no longer maps to a documented Meta crawler.
Meta-WebIndexerAllowSearch indexHonors robots.txtNavigates the web “to improve Meta AI search result quality for users.” Meta: allowing it “helps us cite and link to your content in Meta AI’s responses.” Token:
meta-webindexer. Implied: Meta names only two agents that may bypass robots.txtMeta-ExternalFetcherAllowUser-triggeredPartlyFetches individual links at a user’s request and supports agentic features, “including helping AI navigate websites to complete tasks for users.” Meta: "this crawler may bypass robots.txt rules."
Meta-ExternalAgentYour callModel trainingHonors robots.txtCrawls “for use cases such as training foundation AI models or improving products by indexing content directly.” Meta doesn’t say what blocking it does to Meta AI answers.
FacebookExternalHitAllowOtherPartlyBuilds the title, description and thumbnail preview when a link is shared in Meta’s apps, “such as Facebook, Instagram, or Messenger.” May bypass robots.txt for security or integrity checks
Meta-ExternalAdsYour callOtherHonors robots.txtCrawls “for use cases such as improving advertising and other business-related products and services.” Meta doesn’t connect it to Meta AI answers.
A robots.txt that keeps you in Meta AI search and in link previews while opting out of training looks like this. Named groups override a User-agent: * block, and the rules must be repeated on every subdomain:
# Meta AI search index: allowUser-agent: meta-webindexerAllow: / # User-requested fetches (may bypass robots.txt anyway)User-agent: meta-externalfetcherAllow: / # Link previews in Meta's appsUser-agent: facebookexternalhitAllow: / # Model training: your callUser-agent: meta-externalagentDisallow: / # Ads and business products: your callUser-agent: meta-externaladsDisallow: /Meta’s crawlers are also heavy. Fastly attributed 52% of the AI crawler traffic it saw from mid-April to mid-July 2025 to Meta — more than Google (23%) or OpenAI (20%). Rate-limit if you must, but a 429 to meta-webindexer is a page Meta AI can’t cite.
Meta’s page calls allowlisting by IP address “more secure” than by user agent, but the current version publishes no ranges. Meta’s own network is AS32934; Meta doesn’t document that every crawler request comes from it, but it’s the first check when a meta-* user agent looks spoofed:
# Routes announced by Meta's network (AS32934)whois -h whois.radb.net -- '-i origin AS32934' | grep ^route | headWhere Meta AI's results come from#
Meta AI draws on four pools. You can influence all four, but each has a different door:
| Source | What it supplies | What's documented | Your way in |
|---|---|---|---|
| Search partners | Web results for current questions | Bing (Sept 2023) and Google (Apr 2024) were announced; the 2026 terms name no engine | Rank in Google and Bing |
| Meta’s own index | Pages crawled by Meta-WebIndexer | Meta says it improves Meta AI search and helps it “cite and link” | Allow the crawler; serve clean HTML |
| Licensed news | Real-time news with links to the publisher | Reuters (Oct 2024); CNN, Fox News, USA Today, Le Monde and others (Dec 2025) | Publishers: a licensing deal |
| Meta’s platforms | Public posts, Reels, Groups, Marketplace, the product catalog | Post search from Jan 2025 on (per its tool list); Facebook’s AI Mode (June 2026) | Publish, and get discussed, on Facebook, Instagram and Threads |
Search partners: named once, unnamed now
Microsoft announced in September 2023 that Bing would give Meta AI “more timely and up-to-date answers.” In April 2024 Meta added Google, though Search Engine Journal “could only get Meta AI to search using Bing” in its tests, and Ray-Ban Meta’s release notes described glasses search as “powered in part by Bing.” Since then Meta has stopped naming names. Its AI terms, effective May 13, 2026, say only that Meta “may share certain information with select partners, like search engines.” You can’t optimize for one partner, so being indexed and ranking in both Google and Bing remains the broadest route into the candidate pool.
Meta's own index is growing fast
The Information reported in October 2024 that Meta was building a search engine that crawls the web, to reduce its reliance on Google and Bing (Social Media Today’s summary). Meta-WebIndexer is the visible result. In Promptwatch’s logs it sat near 2.2% of AI crawler requests in mid-July 2026, then became the heaviest AI crawler it tracks, at 37.8% on August 9. Days earlier, Pieter Levels posted that Meta staff told him Meta is “ALLEGEDLY” building its own index so its AI searches don’t end up at Google; Search Engine Roundtable’s report carried no comment from Meta. Treat it as unconfirmed — but the crawl volume is real.
Licensed news gets links
Meta’s first AI news deal, with Reuters in October 2024, came with each such answer expected to “include a link to the Reuters story on which it’s based” (SiliconANGLE). In December 2025 Meta added CNN, Fox News, Fox Sports, Le Monde Group, People Inc., The Daily Caller, The Washington Examiner and USA Today; TechCrunch reported those responses “will also include links to articles.” For a publisher, a deal is the documented route to linked news answers.
Meta's own platforms are a second index
This is what makes Meta AI different. Its tool list includes a “semantic search across Instagram, Threads, and Facebook posts,” built from captions, visual analysis and transcripts, with “Data coverage: posts since 2025-01-01.” At the Muse Spark launch Meta promised “Reels, photos, and posts woven directly into your answers, with credit back to the content creators,” and local answers showing “public posts from locals who know the area.” In June 2026 Facebook added AI Mode, a search tab with answers “grounded in what people are saying publicly across our apps like in Groups and Reels.” Shopping mode searches Facebook Marketplace “alongside options from across the internet.”
Answers inside WhatsApp, Instagram, Facebook and glasses#
Meta AI isn’t one interface. The same assistant answers in a standalone app, a website, three social apps, a messenger and a pair of glasses. Meta’s help center documents one constant: Meta AI “may use sources and links from the internet to inform its responses,” and you can review them by selecting Sources under the response. Beyond that, Meta publishes little about how each surface lays citations out.
| Surface | How people ask | What sources look like |
|---|---|---|
| Meta AI app and meta.ai | A chat with Instant and Thinking modes, plus shopping mode | Links in the answer and a Sources list under the response |
| The Meta AI chat, the search bar above your chats, or by mentioning Meta AI in a group | Documented only generically; Meta publishes no WhatsApp-specific layout | |
| AI Mode in search, and Meta AI on posts in Feed | Answers grounded in public Groups and Reels; web links in AI Mode aren’t documented | |
| Instagram and Threads | DMs and search; @meta.ai mentions in Threads are being tested | Not separately documented |
| Ray-Ban Meta and Oakley Meta glasses | By voice, with “Hey Meta” | Spoken answers; the conversation continues in the app’s history tab |
Two details matter for visibility. Incognito chats, launched on WhatsApp and the Meta AI app in Q2 2026, still search — Meta’s help center says web search in them “begins toggled on” — so conversations even Meta can’t see are part of your audience. On glasses, the answer is heard, not read: since April 2025 the Meta AI app has been the glasses’ companion app, where a conversation started on glasses can be picked up from the history tab. Meta doesn’t document whether sources appear there for spoken answers.
Technical requirements#
Meta publishes no rendering, freshness or structured-data guidance for Meta AI. What’s documented is crawler access, link-preview requirements for facebookexternalhit, and how the search tool behaves in the API. The rest is inference, marked as such.
- Meta-WebIndexer allowedRequired
- In robots.txt and at the WAF. It’s the one agent Meta ties to citing and linking your content in Meta AI.
- Content in the raw HTMLRequired
- Undocumented for
Meta-WebIndexer, but Vercel sawMeta-ExternalAgentfetch pages without running JavaScript. Treat client-rendered text as invisible — see server-side rendering. - Meta-ExternalFetcher not WAF-blockedHelps
- It fetches at a user’s request and may bypass robots.txt anyway. A WAF block only means the answer is built from someone else’s page.
- Fast, error-free responsesHelps
- Meta’s crawl volume makes aggressive rate limits tempting. A
429,5xxor timeout to a Meta agent removes that page from the answer. - The answer near the topHelps
- The tool pattern-matches inside opened pages and passes extracted snippets to the model. A clear sentence early in the page is the easiest thing to find.
- Open Graph in the first 1 MBHelps
- Documented for
facebookexternalhit: OG tags before the first 1 MB, gzip and deflate support, and a response “within a few seconds.” It governs your preview card whenever the URL is shared in Meta’s apps. - Structured dataUnconfirmed
- No statement from Meta. The API hands the model extracted page text, so keep every fact in visible copy, not only in JSON-LD.
- Linked social profilesUnconfirmed
- Meta AI’s post search filters by author.
sameAslinks from your Organization schema to your Facebook, Instagram and Threads accounts can’t hurt; Meta doesn’t document entity matching. - SitemapsUnconfirmed
- Not documented for
Meta-WebIndexer. Keep one with honestlastmodvalues, referenced from robots.txt. - llms.txtUnconfirmed
- Meta publishes llms.txt files for its own developer docs, but hasn’t said its crawlers read other sites’ files.
- IndexNowNo effect
- Meta isn’t on IndexNow’s list of participating engines. It still reaches Bing, a past Meta AI partner.
- NoAI meta tagsNo effect
- Meta says it uses robots.txt “rather than non-standard formats like NoAI tags.”
Test the two agents that matter with Meta’s own documented user-agent strings. The second command is the crawler simulation from Meta’s page:
# Does Meta's search crawler get your real content?curl -s -A "meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)" \ https://yoursite.com/pricing | grep -c "Plans start at"# 0 = text rendered by JavaScript, or a WAF block # Meta's link-preview simulationcurl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \ -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"What Meta AI cites#
Independent citation research on Meta AI barely exists. The large AI-referral panels — SE Ranking’s (101,574 sites, June 2026) and Similarweb’s (September 2026) — don’t break it out at all. The signals below lean on Meta’s own documentation, and say so where they’re our read.
Crawl permission
OfficialMeta says allowing
Meta-WebIndexer“helps us cite and link to your content in Meta AI’s responses” — the only ranking-adjacent statement it has made.Public talk on Meta's apps
OfficialMeta says local, trending and shopping answers draw on public posts and creators across its apps, with “credit back to the content creators.” Facebook’s AI Mode answers from what people say publicly.
Licensed news
OfficialPartner publishers’ articles are linked in news answers by agreement: Reuters, CNN, Fox News, USA Today, Le Monde and others.
Pages that match the rewrite
Our readEach search sends a main query plus alternatives, with an optional date filter. Pages whose titles and opening lines match those phrasings, with a real date, are easier to retrieve.
Classic search rankings
Our readWith partners unnamed and a history of Bing and Google, strong rankings in both are the broadest way into the candidate pool.
A trust gap to close
ObservedIn the Reuters Institute’s six-country survey (Oct 2025), 12% trusted Meta AI against 29% for ChatGPT, and more people distrusted it than trusted it.
The practical upshot: Meta AI rewards the same answer-first, dated pages as other engines, plus something only Meta can search — brand mentions in public posts on Meta’s own apps. A brand nobody discusses on Facebook, Instagram or Threads is missing from the part of Meta AI’s retrieval no other engine has.
Measuring Meta AI#
Meta publishes no webmaster console for Meta AI and doesn’t document the referrer its answer links carry. GA4’s AI Assistant channel names ChatGPT, Gemini, DeepSeek, Copilot and Grok in Google’s channel documentation — not Meta AI. Our read: clicks from the meta.ai website should show meta.ai, while links opened inside Facebook can pass through Meta’s link shim, which reports as l.facebook.com on desktop and lm.facebook.com on mobile — mixed in with ordinary social clicks. Give Meta AI a custom channel, placed above Referral:
# Session source — Meta AI website(^|\.)meta\.ai$ # Meta link shims — Meta AI mixed with ordinary Facebook clicks^(l|lm|m)\.facebook\.com$Your logs are more reliable. meta-webindexer hits show what Meta is indexing; meta-externalfetcher hits are pages fetched for a person’s request — the closest thing to a live-demand signal Meta gives you:
# Meta agent hits by typegrep -oiE "meta-webindexer|meta-externalfetcher|meta-externalagent|meta-externalads|facebookexternalhit" access.log \ | tr 'A-Z' 'a-z' | sort | uniq -c # Pages Meta fetched on a user's requestgrep -i "meta-externalfetcher" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20Finally, test retrieval directly. Meta’s Model API offers a web_search tool whose internals carry the same browser.search, browser.open and browser.find names as the consumer app’s tools, and it can return every result the model considered alongside the subset it cited. Run your target questions through it on a schedule — with the caveat that the app’s prompts, personalization and partners may differ:
import osfrom openai import OpenAI client = OpenAI(base_url="https://api.meta.ai/v1", api_key=os.environ["MODEL_API_KEY"]) r = client.responses.create( model="muse-spark-1.3", input="What's the best project management tool for a five-person team?", tools=[{"type": "web_search"}], include=["web_search_call.results"],) # web_search_call.results = every URL retrieved# url_citation annotations on output_text = the URLs citedMyths worth dropping#
Myth
Blocking Meta-ExternalAgent removes you from Meta AI.
Reality
Meta ties search citations to Meta-WebIndexer. A Meta-ExternalAgent block stops training collection; what it does to answers isn’t documented.
Myth
Meta AI is just Bing.
Reality
Bing was the first partner in 2023 and Google joined in 2024, but Meta names neither today — and it runs its own crawler and searches its own platforms.
Myth
Allowing facebookexternalhit covers Meta AI.
Reality
It only builds link previews. Meta AI search uses Meta-WebIndexer, a separate token.
Myth
NoAI meta tags keep Meta out.
Reality
Meta says it follows robots.txt “rather than non-standard formats like NoAI tags.” Put your rules in robots.txt.
Myth
Meta AI only cites websites.
Reality
It searches Facebook, Instagram and Threads posts too, and Facebook’s AI Mode answers from public Groups and Reels.
The action checklist#
Everything above, in the order we’d do it. Tick items off as you go — your progress is saved in this browser.
0 of 15 done
Meta AI SEO: frequently asked questions#
How do I get my website cited by Meta AI?
Allow Meta-WebIndexer, which Meta says helps Meta AI cite and link to your content, keep your answers in server-rendered HTML, and rank in the classic search engines. Then build a presence on Facebook, Instagram and Threads, whose posts Meta AI also searches.
What search engine does Meta AI use?
Meta doesn’t say anymore. It announced Bing in 2023 and Google in 2024; its May 2026 terms mention only “select partners, like search engines.” It also runs its own crawler, Meta-WebIndexer, and reportedly its own index.
What is Meta-WebIndexer?
Meta’s crawler for Meta AI search. Meta says it “navigates the web to improve Meta AI search result quality” and that allowing it helps Meta AI cite and link to your content. Its robots.txt token is meta-webindexer.
Should I block Meta-ExternalAgent?
Only if you don’t want your content used for training. Meta describes it as crawling for “training foundation AI models or improving products,” and doesn’t say what blocking it does to answers. Keep Meta-WebIndexer allowed either way.
Does Meta AI respect robots.txt?
Mostly. Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalAds are controlled by it. Meta-ExternalFetcher, which fetches links at a user’s request, “may bypass robots.txt rules,” and facebookexternalhit may bypass it for security checks. Changes can take 24 hours to apply.
Can Meta AI read JavaScript-rendered pages?
Meta doesn’t document it. Vercel’s 2024 crawler study found Meta-ExternalAgent doesn’t render JavaScript, so treat text that only appears after scripts run as invisible to Meta AI.
How does Meta AI show sources in WhatsApp?
Meta’s help center describes one pattern for Meta AI: sources you review by selecting Sources under the response. It publishes no WhatsApp-specific layout. Because answers arrive in a chat thread, often a group, your brand name in the sentence matters as much as the link.
Does Meta AI use Facebook and Instagram posts as sources?
Yes. Its tool list includes a semantic search over Facebook, Instagram and Threads posts from January 2025 on, and Meta says answers weave in posts and Reels with credit to their creators. Facebook’s AI Mode answers from public Groups and Reels.
How do I track traffic from Meta AI?
Build a GA4 custom channel for meta.ai, since GA4’s AI Assistant channel doesn’t name Meta AI. In-app clicks may arrive through Facebook’s link shims, mixed with social traffic, so pair analytics with log checks for meta-webindexer and meta-externalfetcher.
Sources
- 1.Meta web crawlersMeta for Developers · developers.facebook.com ↗
- 2.Search groundingMeta Model API Docs · dev.meta.ai ↗
- 3.Introducing Muse SparkMeta Newsroom · about.fb.com ↗
- 4.New AI tools to help you make things happen on FacebookMeta Newsroom · about.fb.com ↗
- 5.Introducing the Meta AI appMeta Newsroom · about.fb.com ↗
- 6.Meta AIs Terms of ServiceMeta · facebook.com ↗
- 7.Report a source or link used in a response by Meta AIMeta Help Center · meta.com ↗
- 8.Toggle web search for incognito chats with Meta AIMeta Help Center · meta.com ↗
- 9.Q2 2026 earnings call transcriptMeta Investor Relations · s21.q4cdn.com ↗
- 10.Q3 2025 earnings call transcriptMeta Investor Relations · s21.q4cdn.com ↗
- 11.Expanding our AI partnership with MetaMicrosoft Bing Blog · blogs.bing.com ↗
- 12.Meta integrates Google and Bing search results into AI assistantSearch Engine Journal · searchenginejournal.com ↗
- 13.Meta's new model is Muse Spark, and meta.ai chat has some interesting toolsSimon Willison · simonwillison.net ↗
- 14.Meta Web Indexer: Meta's crawling surge in AI crawler logsPromptwatch · promptwatch.com ↗
- 15.AI crawlers make up almost 80% of AI bot trafficFastly · fastly.com ↗
- 16.The rise of the AI crawlerVercel · vercel.com ↗
- 17.Meta signs commercial AI data agreements with publishersTechCrunch · techcrunch.com ↗
- 18.Meta inks multiyear AI content licensing deal with ReutersSiliconANGLE · siliconangle.com ↗
- 19.Generative AI and News Report 2025Reuters Institute · reutersinstitute.politics.ox.ac.uk ↗
- 20.Default channel group (AI Assistant)Google Analytics Help · support.google.com ↗
