Home/News/Mueller: skip the markdown layer and the sitemap cache-busting trick

SEO & GEO

Mueller: skip the markdown layer and the sitemap cache-busting trick

2026-09-05 Search Engine Journal SEO & GEO
Mueller: skip the markdown layer and the sitemap cache-busting trick
图:Search Engine Journal

Search Engine Journal's SEO Pulse for September 5 rounded up John Mueller's answers from the week. Two of them address techniques currently making the rounds: serving a separate markdown version to AI crawlers, and hanging a daily-rotating parameter on child sitemap URLs to force search engines to refetch them.

Key facts

  • Effective: Not a rule change — public comments from a Google employee, usable now
  • Who is affected: Self-hosted stores adapting for AI crawlers, and anyone who inherited a "faster crawling" sitemap hack from a big site
  • What to do: Strip rotating parameters from your sitemap index and keep lastmod honest; hold off on a markdown layer until your logs show AI crawlers asking for it

What happened

Someone on r/TechSEO asked whether anyone had seen a major AI bot request markdown, hoping to cut server load by switching. Mueller replied that on his own test sites, "the only crawlers who claim to accept markdown are SEO tools." He noted his server setups do not log the Accept header by default, told anyone curious to set that up and check the numbers first, and closed with "Ymmv."

The second came from Bluesky, where a user had seen a large site attach a fresh parameter to every child sitemap URL each day to force a refetch. Mueller called it "a bad idea because you're signaling that the canonical URL for a piece of content should be constantly changing" — bad for search engines and for your own tracking, when the sitemap protocol's lastmod element already tells crawlers when a file changed.

What this means for independent store owners

Check the second one today. Plenty of WordPress and Shopify SEO plugins — and plenty of agency "crawl acceleration" setups — bolt dynamic parameters onto sitemaps, and nobody looks at them again after install. They will not get you penalized, but they scramble the most basic signal you have, URL stability, and they make your Search Console crawl data hard to read. Open your sitemap index, delete any ?v= or similar daily-changing suffix, and confirm lastmod reflects the real last modification instead of today's date on a timer.

The first one saves you money. Several vendors are now pitching a markdown copy of your content "for AI crawlers," but in Mueller's logs the only things requesting markdown were SEO tools. He is explicit that this is his sample and yours may differ — which is the point. The right move is neither to build it nor to dismiss it, but to log the Accept header for a week or two and see who actually asks for text/markdown. No requests, no content pipeline.

Someone should be watching these changes for you

I run AI automation for independent stores — content, SEO/GEO, product listing, social, and keeping up with policy shifts like this one.

See how it works