Website administrators and digital creators will soon be able to opt out of artificial intelligence training using a single entry in their robots.txt file, bypassing the need to configure separate restrictions for individual tech firms. Filed as a proposed internet standard on October 30, 2026, the shared signal has garnered backing from major technology corporations including Microsoft and OpenAI.
What is the new robots.txt AI training rule?
The proposed mechanism introduces a standardized user-agent designation that informs automated systems whether they can ingest web content for machine learning development. By implementing a specific two-line directive, site owners can request that their pages be excluded entirely from training datasets.
To restrict an entire domain from being utilized for model training, administrators need to append the following lines to their site's configuration file:
User-Agent: __usage_ai-training
Disallow: /
Alternatively, creators can permit specific directories—such as digital press kits or public resources—while blocking the rest of the domain by utilizing an additional allow rule underneath the primary restriction. Common content management systems and SEO plugins typically feature built-in editors to manage these settings easily.
How does this differ from AI answers?
It is essential to distinguish between model training and live AI answers, often referred to as grounding. Training involves feeding web data into an algorithm to build its foundational knowledge base, whereas live answers occur when an AI tool reads a page in real time to answer a user's query, frequently including attribution links.
The new robots.txt standard exclusively governs model training. For live search answers, platforms like Bing rely on separate page- level meta tags. Industry experts note that completely blocking live AI answers may reduce a business's visibility when users seek recommendations through conversational search tools.
When will tech companies adopt the standard?
Implementation timelines vary among major technology platforms. Microsoft announced it will begin honoring the preference on February 1, 2027, during its routine crawling cycles. The corporation will subsequently discontinue support for older legacy tags, such as NoArchive and NoCache, for AI purposes after March 1, 2027.
OpenAI has indicated that support will roll out over the following months, while Google has pointed toward upcoming adoption timelines. The draft standard was authored by Kevin Kelley and submitted to the Internet Engineering Task Force, the body responsible for developing and promoting internet standards. As an individual draft, it currently serves as a voluntary protocol rather than a legally mandated requirement.