Apple Quietly Tripled Applebot's Crawl Capacity Weeks Before Siri AI Ships

Apple does not announce crawler changes. It edits a JSON file and lets you find out.

Late in July, the file listing Applebot's published IP ranges grew by 21 new CIDR prefixes: eighteen /24 blocks and three /28 blocks, all inside 17.166.0.0/16. That is 4,656 new addresses. The published pool went from 2,400 to 7,056, an increase of roughly 194% in a single edit, detected by monitoring on 17 August (PPC Land, Search Engine Roundtable).

No blog post. No developer note. No explanation of any kind.

Why the timing matters

Apple confirmed the rebuilt Siri at WWDC in June. It runs on the next generation of Apple Foundation Models, it gets its own app, and it can pull current information from the web to answer open ended questions (Apple Newsroom, TechCrunch). iOS 27 lands in September.

You do not triple the addresses you crawl from because you plan to crawl the same amount from more places. Infrastructure gets provisioned ahead of load.

We are reading this as capacity being staged for a product launch, and we will say plainly that this is inference rather than fact. Apple has said nothing, and a large IP allocation can also mean a routing change or a data centre migration. But the size of it, the concentration in one /16, and the four weeks between the file edit and the iOS 27 release window all point the same direction.

The mistake we see most: blocking the wrong user agent

Here is the part worth acting on regardless of what Apple's plans turn out to be.

Apple runs two user agents and they do completely different jobs. Most sites treat them as one thing.

Applebot is the crawler. It fetches your pages and feeds Spotlight, Siri and Safari search. It is the reason your site can appear in an Apple search surface at all.

Applebot-Extended does not crawl anything. It is a control token. Its only purpose is to tell Apple whether the content it already fetched can be used to train foundation models (Apple).

So these two rules are not variations on a theme:

User-agent: Applebot-Extended
Disallow: /

That says: index me for search, do not train on me. It is the publisher opt out, and plenty of large publishers have taken it, including The New York Times, the Financial Times and Condé Nast.

User-agent: Applebot
Disallow: /

That says: remove me from Apple search entirely. Different decision, much bigger consequence, and we run into sites that made it by accident.

The accident usually happens one of two ways. Someone read a 2024 thread about blocking AI crawlers and pasted a list of user agents into robots.txt without checking what each one did. Or the site sits behind a managed bot rule that treats every non-Google crawler as scraping. We wrote about the second pattern in Cloudflare's AI crawler blocking, and Applebot is one of the agents that quietly gets caught in it.

The Googlebot fallback almost nobody knows about

This one surprises people, and it is in Apple's own documentation.

If your robots.txt has no rules addressing Applebot specifically, Applebot follows your Googlebot rules instead.

Read that again with your own robots.txt in mind. Every restriction you ever wrote for Googlebot, including the crawl trap exclusions and the faceted navigation blocks and the staging path you disallowed in 2021, is currently governing what Apple can see. If you have a tight Googlebot block anywhere, you inherited it for Siri without deciding to.

That is defensible when your Googlebot rules are current and deliberate. It is a problem when they are archaeology. Most robots.txt files we audit are archaeology.

Your IP allowlist just went stale

If you verify crawlers by matching against a hardcoded list of Apple IP ranges in your WAF or your log pipeline, 4,656 addresses now fail that check.

Depending on how the rule is written, real Applebot traffic from those addresses gets rate limited, served a challenge, or logged as a spoofed bot. None of those outcomes are visible unless someone goes looking.

Apple documents two verification methods, and only one of them survives an event like this. Reverse DNS resolves genuine Applebot traffic to the *.applebot.apple.com domain, and a forward confirm on that hostname is the check that keeps working when the address pool changes. Matching against the published CIDR list only works if something re-fetches that file on a schedule.

Our position on this is not Apple specific. Verify crawlers by reverse DNS, not by a list you copied into a config file. Every major operator expands ranges without warning, and a static allowlist is a slow leak you will not notice until traffic is already gone. The same logic applies to the AI crawlers we covered in the robots.txt guide.

Our take: Apple is the surface nobody is optimising for, and that has been rational until now

We will frame this clearly as opinion.

Almost no business has ever done a single piece of work aimed at Apple's search surfaces, and until this year that was the correct allocation of effort. Siri could not answer an open web question. Spotlight was a launcher. There was nothing to optimise for, so nobody optimised.

What changes in September is that the assistant on roughly half the phones in the US and a large share of phones in Cyprus becomes something that reads the web and answers from it. Combine that with Assistant being retired from Android on 4 September and both default phone assistants become language models within the same few weeks.

The honest version of the opportunity is smaller than the headline. Nobody is going to build an Apple specific content strategy, and they should not. Apple crawls the open web with an ordinary crawler and reads ordinary pages. The work that makes you retrievable by ChatGPT and AI Mode is the same work that makes you retrievable by Siri.

The difference is the failure mode. With Google you have Search Console telling you when something breaks. With Apple you have nothing. There is no report, no coverage view, no notification. If Applebot cannot reach you, the only place that fact exists is in your server logs, and it will sit there for a year unless someone reads them.

That asymmetry is the whole argument for spending twenty minutes on this. The upside is ordinary. The downside is invisible.

What to check this week

Five things, none of which take long.

Grep your logs for Applebot. Confirm it is actually reaching you and getting 200s. If you have never seen it in your logs, that is the finding.

Read your robots.txt properly. Find every rule that touches Applebot or Applebot-Extended, and if there are none, go read your Googlebot rules instead, because those are the ones in force.

Decide the training question deliberately. Blocking Applebot-Extended is a legitimate choice and it costs you nothing in search visibility. Just make it on purpose rather than inheriting it from a pasted list.

Check your WAF and CDN rules. Managed bot protection is where good crawlers go to die quietly. Look at what your challenge and rate limit rules do to a request from an Apple range.

Switch crawler verification to reverse DNS. If anything in your stack allowlists Apple by hardcoded IP, that list is now missing 4,656 addresses.

None of this is glamorous, and that is roughly the point. The sites that show up in new answer surfaces are usually not the ones with a clever strategy for them. They are the ones that were reachable when the crawler arrived.

If you want a second pair of eyes on how your site handles AI and search crawlers, that is standard scope in our technical SEO and AI search optimisation work, and it is one of the first things we look at in a free SEO review.