AGENTS.md Is the New robots.txt

Ripper·May 4, 2026·6 min read

Every coding agent that has touched my repos in the last three months has done the same thing: it opens the root directory, looks for a file named AGENTS.md, reads it before anything else, and then starts working. Claude Code does this. Cursor does this. Cline does this. The agent ecosystem has converged on a convention that almost nobody knows exists.

AGENTS.md is the new robots.txt. Different audience, same idea: a small text file at your repo root that tells automated visitors how to behave. Robots.txt told search crawlers what they could index. AGENTS.md tells coding agents how to write code in your project so they do not break things. The difference is that AGENTS.md is much more useful, because the audience is much smarter, and the cost of getting it wrong is much higher than a missing search hit.

Why Agents Read It First

When an agent opens a fresh repo it has zero context. It does not know which package manager you use, which test command works, which directories are vendored, which files are auto-generated and should never be touched, or what your style preferences are. It can guess from package.json or pyproject.toml or a Makefile, but guessing burns tokens and produces worse output.

AGENTS.md is the cheat sheet. The agent reads thirty lines, gets oriented in two seconds, and writes correct code on the first attempt instead of spending ten minutes pattern-matching its way through your conventions. That difference shows up in the diff and in the time-to-first-useful-PR.

For my own projects, I can feel the difference between repos that have an AGENTS.md and repos that do not. The ones with AGENTS.md get accurate, in-style code. The ones without get generically reasonable code that often fails my pre-commit hook because the agent guessed wrong about something the human team already settled.

What to Put In One

The file is freeform Markdown. There is no schema. But the same handful of sections show up across every good AGENTS.md I have seen:

What this project is. One sentence. The agent will infer almost everything else from this. Do not skip it.
How to install dependencies. The exact command. Not "run npm or yarn or whatever you prefer." Pick one. The agent will use whatever you list.
How to run tests and lint. The exact commands again. Agents tend to be conservative; if you tell them "npm test" runs the suite, they will run it before submitting a PR.
Code style rules that humans cannot infer from a linter. Naming preferences, file layout, what counts as a small commit, whether you use semicolons in TypeScript. The stuff your team agreed on but never wrote down.
Don't-touch zones. Auto-generated files, vendored dependencies, lockfiles you regenerate via a specific command. Agents will respect these if you list them. They will edit them if you do not.
Where to find more detail. Pointers to docs/ARCHITECTURE.md, the README, internal wikis. Agents follow links.

You do not need a section for things the agent can read off the file tree. You do not need to explain that you use TypeScript if there is a tsconfig.json sitting there. The point of AGENTS.md is to capture what the agent cannot infer.

The Discoverability Question

Robots.txt works because search engines all agreed to look for it at a specific path. The same convergence is happening with AGENTS.md, but it has happened informally and quickly, without anyone publishing a spec.

OpenAI's Codex, Anthropic's Claude Code, Cursor's background agents, Cline, Aider, GitHub Copilot Agent: every major coding agent I have tested in 2026 reads AGENTS.md if it exists. Some also read CLAUDE.md or CONTRIBUTING.md as fallbacks. None of them read random other filenames. The path is the standard now.

That convergence is not because everyone copied each other. It is because AGENTS.md is the first thing an agent thinks of when it lands in a strange repo. The name is good. The location is obvious. The format is freeform enough to write in five minutes. So everyone reaches for it independently and the convention sticks.

An Example Worth Stealing

Here is a thirty-line AGENTS.md that would meaningfully improve any Node project:

# AGENTS.md

This is a [one sentence about what the project is].

## Install
npm install

## Run tests
npm test

## Run lint
npm run lint

## Run dev server
npm run dev

## Code style
- TypeScript strict mode, no `any` types.
- Tailwind only for styling. No CSS modules.
- Functional React components, hooks only.
- Use `@/` path alias for imports from src/.

## Don't touch
- src/generated/  (regenerate via npm run codegen)
- public/api/     (regenerate via npm run prebuild)
- lockfiles       (always commit them)

## Conventions
- Commits are imperative present tense.
- PR titles match the conventional commits format.
- Comments only when the WHY is non-obvious.

## More detail
- docs/ARCHITECTURE.md  (full system reference)
- README.md             (user-facing intro)

That is it. Thirty lines. Ten minutes to write. Every agent that visits your repo from now on opens that file before doing anything else.

Why You Should Ship One This Week

Agent traffic is not a small slice anymore. On TensorFeed I see machine-readable feed pulls outpacing human page views. On a public repo, AI assistants reading the source for users they will never see is becoming the dominant traffic pattern. You are writing code for both audiences whether you planned to or not.

The repos that ship a clean AGENTS.md will quietly outperform the repos that do not. Their PRs will be cleaner. Their issue triage will be faster. Their forks will be more useful. And the kicker: future agents will remember which repos respected them. Tool descriptions, embeddings, whatever the next architecture is, the friction-of-a-fresh-repo-without-AGENTS.md is going to feel like the friction of a website without a sitemap. A small thing that the wrong repos still skip.

The TensorFeed Take

We ship an AGENTS.md at our project root (we use the CLAUDE.md naming, but it is the same idea, and most agents that read AGENTS.md also read CLAUDE.md). It contains the rules of the codebase, the do-not-touch zones, and the brand voice rules. Every PR an agent makes against TensorFeed already passes the lint, fits the file conventions, and obeys the "no em dashes" rule. Not because the agent is psychic. Because we wrote it down.

If you maintain a public repo and agents are touching it, take fifteen minutes this week and write an AGENTS.md. The yield is huge. The cost is nothing.

For more on building for AI agents, see Why Every Developer Needs an llms.txt File and Building for AI Agents.