AI agent readiness

robots.txt answers exactly one question: may a bot open this page. But "may it read" and "may it train a model on this" are entirely different things, and you have never answered the second one. Here you answer three questions separately: may models be trained on your content, may you appear in search, and may your text be retold inside chatbot answers. The usual business call: allow search and answers (they bring customers), refuse training.

—

Add a line to robots.txt like "Content-Signal: ai-train=no, search=yes, ai-input=yes". Here ai-train is model training, search is appearing in search, ai-input is your text being retold in an AI answer. yes allows, no refuses. The line applies to the User-agent block it sits in.

To a person your page is a menu, banners, pop-ups and some text in the middle. An AI assistant receives all of it mixed together and has to guess which part matters. It guesses wrong exactly where it costs most: it takes the struck-through price, or the heading of the block next door. Serve the same material as plain text on request and the ambiguity disappears. Ordinary visitors notice nothing — they still get the normal page.

—

Teach the server to tell who is asking. When a request carries "Accept: text/markdown", return the text version and label it text/markdown. Always add "Vary: Accept" — otherwise a cache will one day hand the bare text to a live visitor.

This is the first thing any bot asks for. Until the file exists you control nothing: staging sections, service pages and drafts are open to everyone. A particular nuisance is serving an ordinary error page at this address instead of a file — the bot reads the rules as broken and crawls everything, as though you had forbidden nothing at all.

—

Put robots.txt at the site root so it opens at yoursite.com/robots.txt. Inside: User-agent blocks with Allow and Disallow rules, plus a Sitemap line with the full sitemap address. Serve it as plain text. Check that unknown addresses return a 404 rather than quietly serving the homepage.

A sitemap is a ready-made inventory of everything you have. Without one an assistant follows links and sees only what someone already linked to. New things suffer most: a fresh product, a recent article, a page buried one link deep in a menu. Formally they are on the site; for the assistant they do not exist.

—

Create sitemap.xml listing all public pages and declare its full address in robots.txt with a "Sitemap:" line. Past 50,000 pages, split it into several files behind one index. Nearly every CMS can do this for you — often it is a setting or a plugin away.

robots.txt usually holds a single rule "for everyone" — the asterisk. It catches the search engine, the image scraper and the AI bot all at once. Which means you never made a decision about AI: one happened by default. And the decisions differ. A bot that brings you customers out of a chatbot is welcome. A bot hoovering up your catalog to train someone else's model, less so. You can only tell them apart by naming them — and those names the vendors do honour.

—

Add separate User-agent blocks to robots.txt for specific bots with their own rules. The main ones: GPTBot (OpenAI training), OAI-SearchBot and ChatGPT-User (showing and referring from ChatGPT), ClaudeBot, Google-Extended (Google training), PerplexityBot, CCBot. Keep the asterisk rule last as a fallback.

This one matters only if you have a public API — a way to get your data programmatically. Then an assistant needs to learn it exists and where its description lives. Otherwise it scrapes data off pages while an exact, ready-made route sits right there. No API? Skip this without a second thought.

—

Publish a small file at /.well-known/api-catalog with links to your API description and its documentation.

An assistant does not always download the whole page — often it asks only for the service part of the response, where the server briefly describes itself. Put pointers to what matters there — a description, a catalog, a feed — and even those who never parse the page will see them. It is the cheapest way to say: look over here.

—

Configure the homepage to send Link headers pointing at what matters: describedby (description), service-desc (API description), alternate (other versions), api-catalog. This is done at the web server level, no application code involved. Plumbing links like stylesheets and icons do not count — everyone has those and they tell an assistant nothing.

This matters if you have customer accounts and an assistant should act inside one on a customer's behalf: check orders, renew a service, download documents. A normal help page will not do — it is written for a human and scattered across sections. What is needed is a short text at a fixed address saying, point by point: register like this, sign in like that, these are the limits.

—

Put an auth.md file at the site root. In plain text describe: how to register, how to sign in, whether access keys are needed and where to get them, how many requests per minute are acceptable. Check that the file is served as text and not as a styled page.

The machine-readable sequel to the previous item. For an assistant to sign in on a customer's behalf it needs the technical addresses of your login system. Either a developer types them in by hand for every integration, or you publish one file and the connection configures itself. Only worth doing if you genuinely have a login system — the file alone opens nothing.

—

Publish /.well-known/oauth-authorization-server (or openid-configuration) with your login system's addresses and the required issuer field.

An assistant should know up front which pass a restricted area needs. Otherwise it works it out by trial: knocks, gets refused, tries another way. For you that is wasted load and odd spikes of rejections in the logs; for the customer, an assistant that "somehow cannot get in".

—

Publish /.well-known/oauth-protected-resource listing which areas are restricted and which passes work for them. Pairs with the previous item.

This item and the next two matter only if you run your own AI assistant on the site — a chatbot, a consultant, a product picker. Then other assistants need to find it and understand what it can be asked. The card is its profile: what it is called, what it does, where to reach it. No assistant of your own? Skip all three.

—

Publish /.well-known/agent-card.json with your assistant's name, description, capability list and contact address.

The card says who your assistant is. The skills list says what it concretely does: pick a product, quote shipping, process a return. Without it another assistant sees a counterpart but has no idea what to come to it with — so it does not come.

—

Publish a list of skills: each with a name, a clear description, and the data needed to call it.

The deepest level: instead of reading your site, a model calls your functions directly — "find the product", "check stock", "create a request". The data is always right because it comes from your system rather than being read off a page. This is real development work: a running server is required, a file is not enough.

—

Stand up an MCP server with the functions you want to expose and publish its card at /.well-known/mcp.json.

This item is not about who visits you but about who you send out: your own scraper, price monitor, aggregator. Publishing your keys lets other sites confirm a request really came from you rather than from someone hiding behind your name. If you run no bots, this does not apply — it is not a shortcoming.

—

Publish a key set at /.well-known/http-message-signatures-directory and sign your bots' outbound requests.

An assistant working inside your customer's browser currently pokes at your markup blindly: hunting for the search field, guessing at the button. Declare your site's actions explicitly — "search", "filter", "add to cart" — and it starts using the site deliberately instead of guessing clicks.

—

Declare the main interface actions as tools for an assistant. Start with search and filtering — the most common flows.

A way to announce your AI assistants in the domain settings so they can be found before anyone visits the site. Faster than a file, with one condition: without a signed domain, anyone on the path of the request can forge that record. So signing is not a nicety here, it is part of the job.

—

Add TXT records with your assistants' addresses in your domain settings and turn on DNSSEC — the domain signature. Requires access to your domain control panel.

When a customer asks an assistant to "find where it is cheaper", it walks the shops collecting prices off pages. That is slow, and it gets things wrong exactly where it must not: it takes the struck-through price, or misses that the item is out of stock. An open catalog fixes that — the assistant receives your range, prices and stock ready-made, straight from your system.

—

Publish /.well-known/acp.json with your catalog addresses and the rules for using it. Only meaningful with a live product database behind it.

The step after the catalog: the assistant not only found the product but carried it through to an order. The hard question here is not technical but one of trust — who confirms the purchase and when, so a customer is never charged for something they did not ask for. This is declared not as its own file but as a section inside your assistant's card.

—

Add a payment section to the assistant card and state your role — merchant, for example. You need the card itself first.

Fits if what you sell is access rather than goods: exports, calculations, calls to your service. The assistant invokes the function and pays for it on the spot — nobody is present at the moment of the transaction. Useful when the price is pennies per call and issuing an invoice by hand makes no sense.

—

Describe the payment terms in your API's technical description and install payment handling. You will need to connect it to your payment provider.

If some of your material is paid, an assistant has exactly two options: leave empty-handed, or look for a way around. Both are bad. A third appears once there is a clear rule: here is the price, here is how to pay, here is what opens afterwards. Then gated content starts earning from this side too.

—

Publish /.well-known/ucp.json with your access terms and payment flow. It needs working payment collection behind it, or it stays a statement of intent.

There is no description file here — the rule lives in how the server behaves. To a paid request it answers "payment required", the assistant pays and repeats the request. Fits when the price is per call rather than per month: one-off calculations, lookups, exports on demand.

—

Install a handler that answers protected addresses with status 402 and a payment header. You will need a facilitator address and a wallet.