RadarShare

Meta pauses updated advertiser terms that would have expanded crawling of brand websites

The update would have broadened how Meta crawls advertiser sites. The current rules stay in place, and they already let Meta use that content for research and training.

By MOBAEO Newsroom··3 min read

Follow
Creative installation of rocks suspended from strings, casting striking shadows.
Photo: Jimmy Chan/Pexels

Meta has paused an update to four sets of business and advertiser terms that was due to take effect on November 20, 2026: the Commercial Terms, the Product Catalog Terms, the Self-Serve Ad Terms and the Generative AI Terms for Ad Creatives. Until further notice, the current versions remain in force, and they already allow the company to crawl advertiser websites.

According to Search Engine Journal, in a report published on November 10, a notice in the Meta Business Help Center says the updates "are paused and will not take effect on November 20, 2026." Each document's page now opens with a line stating that the changes are on hold.

Nieman Lab reported the notice on November 5 and spoke with a Meta spokesperson: "We're pausing the changes as we continue to engage with stakeholders and take the time needed to get this right." The company has not set a timeline for resuming the update, nor has it said how much of the revised text will change.

What applies today

The Commercial Terms currently in force, dated March 6, 2024, contain no paragraph on crawling. The paused version would have added one, covering sites connected to a business's products or services. These terms apply to any commercial use, including managing a Facebook Page.

The Self-Serve Ad Terms, dated August 22, 2025, do provide for crawling. By placing an ad order, the advertiser grants Meta the right to "access, index, cache, analyze or crawl" the content of its domains. The listed uses include developing advertising and other commercial products, "research and training," and the integrity of Meta's products. These terms make no mention of robots.txt.

The Generative AI Terms for Ad Creatives, dated May 6, 2024, also allow Meta to crawl the URLs advertisers provide, for use in the ads themselves.

The AI features section is on hold too

In October, Adweek reported that the paused Commercial Terms included a new section called AI Features. Under it, the advertiser would be "solely responsible" for all outputs and changes produced by Meta's AI features, including those the company turns on automatically. Under the current terms, that responsibility depends on whether the advertiser uses or publishes the generated content. A Meta spokesperson told Adweek that Advantage+ features can be turned off one by one, at any time, in Ads Manager.

Robots.txt, the courts and a regulatory probe

Meta says it has not changed how it crawls and that robots.txt controls remain the same. The company's documentation identifies Meta-ExternalAds, used for ads and other commercial products, and Meta-ExternalAgent, used, for example, to train foundation AI models. The same page says two other Meta crawlers may ignore robots.txt in certain situations.

According to Nieman Lab, courts in Brazil and Norway granted injunctions against these terms following complaints from news organizations. In the UK, the Competition and Markets Authority has an open investigation into both the current terms and the proposed changes, with the first information-gathering phase expected to run until January 2027, a timeline that may be revised.

For brand teams, the pause changes little day to day: every new ad order still grants Meta the right to crawl your site, including for research and training. Since Meta's documentation includes robots.txt rules for Meta-ExternalAds and Meta-ExternalAgent, it is worth reviewing how your site's file handles each of them.

Key questions

Do Meta's new terms take effect on November 20?

No. Meta has paused the update, and the current terms remain in force until further notice, with no set date for any change.

Does advertising on Meta give the company the right to crawl a brand's website?

Yes. Under the Self-Serve Ad Terms dated August 22, 2025, every ad order authorizes Meta to crawl the content of the advertiser's domains, including for research and training, and those terms make no mention of robots.txt.

Can you control Meta's crawlers with robots.txt?

Meta says its robots.txt options remain the same, and its documentation includes rules for Meta-ExternalAds and Meta-ExternalAgent. The same documentation notes that two other Meta crawlers may ignore robots.txt in certain situations.

Source: Search Engine Journal.