--- title: "How to Make Your Website AI-Friendly (What Actually Works in 2026)" url: "https://michalak.world/posts/how-to-make-your-website-ai-friendly/" description: "Making a site AI-ready comes down to four things that are actually enforced: let AI crawlers through at the network edge, name them in robots.txt, describe yourself as a resolvable entity in Schema.org JSON-LD, and keep your sitemap and modification dates honest. Most other AI SEO advice is unmeasurable." summary: "A practical, evidence-ordered guide to making a website AI-friendly, based on rebuilding this site for AI discoverability: what AI crawlers actually enforce, why structured data is the real lever, and which popular AI SEO tactics are not worth your time." date: 2026-08-17 lastmod: 2026-08-19 tags: ["AI SEO","SEO","Structured Data","Schema.org","Cloudflare","Hugo","AI Consulting","AI Agents","Technology Strategy"] section: "posts" author: "Michael Michalak" site: "https://michalak.world/" --- # How to Make Your Website AI-Friendly (What Actually Works in 2026) Making a site AI-ready comes down to four things that are actually enforced: let AI crawlers through at the network edge, name them in robots.txt, describe yourself as a resolvable entity in Schema.org JSON-LD, and keep your sitemap and modification dates honest. Most other AI SEO advice is unmeasurable. ## Key takeaways - **Access comes first.** If your CDN or WAF blocks AI bots at the network edge, nothing else you do matters — a blocked crawler never even reads your `robots.txt`. - **Name each bot explicitly in `robots.txt`.** Every major operator runs several crawlers for different purposes, and allowing one does nothing for the others. - **Structured data is the highest-leverage change.** An AI system can only cite you as an entity if you give it a resolvable one: a `Person` or `Organization` with a stable `@id` and `sameAs` links it can verify. - **Freshness signals have to be true.** Most sites emit a `dateModified` that just restates the publish date, which tells an AI system nothing. - **`llms.txt` is close to a no-op.** An Ahrefs study of 137,000 domains found 97% of `llms.txt` files got zero requests in May 2026. Ship it because AI coding agents read it, not because it will win you citations. - **Write answer-first.** Question-and-answer pairs and standalone opening summaries are the most extractable content shapes for an answer engine. - **`auth.md` is not an SEO file.** It tells an agent how to *register* for a service that issues credentials. If you only publish content, the honest implementation is a short file saying no registration exists. "AI SEO" has become a product category, and most of what is sold under that label cannot be measured. I spent August rebuilding this site for AI discoverability, and the useful part of that work was not the tactics — it was sorting the recommendations that are actually *enforced* by AI crawlers from the ones that just sound plausible. Here is what survived that sort, in the order that matters. --- ## First, understand the three ways AI reaches your content Every major AI operator runs **several** crawlers with different jobs, and they are governed independently. This is the detail that trips up most site owners: blocking or allowing one does nothing to the others. | Purpose | What it does | Examples | | --- | --- | --- | | **Training** | Collects content to train or fine-tune models | `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `meta-externalagent` | | **Search / answer index** | Indexes content to answer questions later, with citations | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` | | **User-initiated fetch** | Fetches a page in real time because a person asked | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | The distinction is a business decision, not a technical one. Training gets you into the model's weights with no attribution and no referral traffic. Search and user-initiated fetches are what produce a **cited link back to you**. If you want to be recommended when someone asks a general question, allow all three. If your content *is* your product, block training and allow the other two. --- ## Step 1: Confirm you are not already blocking them This is the step that gates everything else, and it is the one people skip because it lives in a dashboard rather than in the codebase. A block at your CDN or WAF happens at the network level. The crawler gets a `403` and **never reaches the `robots.txt` you carefully wrote**. All the structured data in the world is worthless behind that. If you are on Cloudflare, this is time-sensitive. On **September 15, 2026**, Cloudflare applies new default AI bot policies: bots classified as Training or Agent get blocked on pages displaying ads, and mixed-purpose crawlers that combine Search and Training are blocked by any configuration that blocks training. Free-plan customers who have never touched these settings are moved onto the new defaults automatically. Two settings deserve specific attention: - **Managed `robots.txt`** — when enabled, Cloudflare serves its *own* `robots.txt` containing `Disallow` rules for known AI crawlers plus a Content Signals policy defaulting to `ai-train=no`. That silently overrides whatever your site generates. - **The legacy "Block AI bots" toggle** — deprecating on the same date, and it now also blocks mixed-purpose crawlers. That includes Googlebot. Test it from outside, with a real bot user agent. All of these should return `200`: ```bash curl -sI -A "GPTBot/1.0" https://example.com/ | head -1 curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -1 curl -sI -A "ClaudeBot/1.0" https://example.com/ | head -1 ``` A `403` means your edge configuration is the problem, and no amount of on-site work will fix it. --- ## Step 2: Name each bot in `robots.txt` A bare `User-agent: * / Allow: /` is technically sufficient. Naming each crawler is still worth doing, because it makes your policy explicit, self-documenting, and robust against wildcard-matching quirks in individual crawlers. ``` User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=yes Sitemap: https://example.com/sitemap.xml ``` That `Content-Signal` line is Cloudflare's Content Signals Policy, which expresses intent separately for search indexing, real-time answer generation (`ai-input`), and model training. Unlike much of this space, `robots.txt` compliance is genuinely documented by every major operator — OpenAI, Anthropic, Google, and Perplexity all publish that they honor it. One warning if you use a static site generator: many themes deliberately emit `Disallow: /` in development builds. Check the file your **production** build produces, not the one your local server serves. --- ## Step 3: Structured data — the real visibility lever This is where most sites leave the largest gain on the table. Nearly every CMS and theme emits something like this: ```json "author": { "@type": "Person", "name": "Michael Michalak" } ``` An AI system cannot do anything useful with that. It is a **name string**, not an entity. There is nothing to resolve it against, nothing to verify, and no relationship between the author of an article and the person the site is about. What you want instead is a single JSON-LD `@graph` with cross-referenced `@id` values, so every node on every page points at the same canonical entity: ```json { "@context": "https://schema.org", "@graph": [ { "@type": "Person", "@id": "https://example.com/#person", "name": "Michael Michalak", "jobTitle": "Backend Developer, Fractional CTO, AI Consultant", "sameAs": [ "https://www.linkedin.com/in/michaelmichalak", "https://github.com/mikemichalak" ], "knowsAbout": ["Drupal", "Drupal Commerce", "AI Agents"], "alumniOf": { "@type": "CollegeOrUniversity", "name": "Purdue University" } }, { "@type": "WebSite", "@id": "https://example.com/#website", "publisher": { "@id": "https://example.com/#person" } }, { "@type": "BlogPosting", "author": { "@id": "https://example.com/#person" }, "publisher": { "@id": "https://example.com/#person" } } ] } ``` The `@id` values are the whole point. They turn a set of unrelated per-page snippets into one graph that consistently says *this article was written by this specific person, who is the subject of this site, and who is also these two external profiles.* Two things to get right: - **Use the type that matches the page.** A page about a person should be `ProfilePage` with `mainEntity` pointing at your `Person`. Pages describing what you offer should be `Service`, not a generic `Article`. Add `BreadcrumbList` so the hierarchy is explicit. - **Make `sameAs` reciprocal.** `sameAs` is only a *claim* until the external profile links back. Put your site URL in your LinkedIn and GitHub profile fields. Unverifiable claims in structured data are worse than absent ones. Keep the facts in one data file rather than scattered through templates, and only assert things a human can confirm from visible page content. Validate with the [Google Rich Results Test](https://search.google.com/test/rich-results) and the [Schema.org validator](https://validator.schema.org/) after deploying. --- ## Step 4: Make your freshness signals true Check your own site for this one, because it is almost universal and completely invisible. Across the 76 pages here, **zero** set a modification date. Hugo silently fell back to the publish date, which meant every `dateModified` in the structured data and every `` in the sitemap was just restating when the page was first written. A page I had substantially revised looked untouched since 2019. In Hugo the fix is `enableGitInfo = true`, which derives modification dates from the last commit that touched each file. Every generator has an equivalent. The catch worth knowing: this needs **full git history at build time**. Most hosts, including Cloudflare Pages, clone with depth 1 by default. If all your `lastmod` values collapse to the build date, that is why. While you are in there: - **Create real section landing pages.** `/posts/` and `/experience/` were rendering untitled and undescribed here, and both are prime targets for category-level queries. - **Submit your sitemap** to [Google Search Console](https://search.google.com/search-console) and [Bing Webmaster Tools](https://www.bing.com/webmasters). AI answer engines lean on conventional search indexes far more than on anything AI-specific, and Bing's index still feeds several AI answer products. --- ## Step 5: Shape content so a passage can stand alone An answer engine does not cite pages. It extracts **passages**. Content that requires reading three sections to assemble one answer does not get used. - **Fix your heading hierarchy.** If your generator renders the front matter title as the `H1`, a `#` heading in the body creates a second one and flattens the structure parsers use to segment a page. Seven pages here had this, one of them using `#` for every single section. - **Add question-and-answer pairs.** Q&A is the single most extractable shape there is. Pair the visible content with `FAQPage` JSON-LD generated from the *same source*, so the markup can never describe a question a visitor cannot see — which is what Google requires. - **Lead with the answer.** Put a complete, standalone summary in the opening block so an extractive summarizer finds a finished answer instead of having to synthesize one. - **Write descriptions that add information.** A `description` that restates the title gives an AI system nothing it did not already have from the heading. Write it as a self-contained answer to the question the page addresses. --- ## Step 6: Machine-readable endpoints, with realistic expectations Two things here, and they are not equally worthwhile. **Markdown twins of every page: worth it.** Hugo 0.164 ships a built-in `markdown` output format, so every page gets a clean `.md` version at `/posts/slug/index.md`, discoverable through the `` tag. An agent gets source Markdown with its lists, code blocks, and heading structure intact, rather than HTML that has been crudely stripped of tags. **`llms.txt`: near-zero value, near-zero cost.** I want to be direct about this because the file gets recommended constantly: - An Ahrefs study of 137,000 domains found **97% of `llms.txt` files received zero requests** in May 2026. - Google stated in June 2026 that the file is **not required** for Search. - OpenAI's and Anthropic's crawler documentation both point site owners to `robots.txt` instead. Ship it anyway if it costs you one build step, because AI coding agents like Cursor, Claude Code, and Copilot *do* read it. Just do not expect citations from it, and do not let anyone sell you an AI SEO package whose centerpiece it is. --- ## A note on `auth.md`, which is not an SEO file One more file has entered this conversation recently, and it belongs in a different category from everything above. [`auth.md`](https://github.com/workos/auth.md) is an open protocol WorkOS published in May 2026. It answers a question none of the other files touch: how does an AI agent *sign up* for your service on behalf of a user, without a human filling in a form? The Markdown file at `https://yourapp.com/auth.md` is the prose entry point. The actual discovery path is two hops of existing OAuth standards: 1. Your API returns `401` with a `WWW-Authenticate: Bearer resource_metadata="…"` header. 2. That points at Protected Resource Metadata ([RFC 9728](https://www.rfc-editor.org/rfc/rfc9728)) at `/.well-known/oauth-protected-resource`, naming the resource, its scopes, and its authorization server. 3. The authorization server's metadata ([RFC 8414](https://www.rfc-editor.org/rfc/rfc8414)) at `/.well-known/oauth-authorization-server` carries an `agent_auth` block with the registration endpoints and the flows you accept. There are two flows. In **agent verified**, the agent's identity provider signs an ID-JAG assertion vouching for the user, and the service issues a credential synchronously with no human in the loop. In **user claimed**, the agent shows the user a one-time code to confirm, which needs no provider integration at all. **Whether this applies to you is a yes-or-no question, not a spectrum.** If your site issues credentials — accounts, an API, anything sitting behind a `401` — it is worth reading properly. If you publish content, it is not for you, and publishing the discovery documents anyway means advertising endpoints that `404`. That is the same failure as an unverifiable `sameAs`: a machine-readable claim that does not survive being checked. This site is squarely in the second group, so I published an `/auth.md` that says exactly that — no registration exists, no credential is needed, here are the open endpoints, do not go looking for a `register_uri`. An agent gets its answer in one fetch instead of walking a discovery chain to a dead end. One safety note for anyone building scanners: **do not probe `POST /agent/auth` to test whether a site supports this.** That is a registration endpoint. Hitting it can create accounts, send email, and issue live credentials. The public discovery documents are the safe source of truth, and they are the only thing worth reading unattended. --- ## How to tell whether it worked There is no rank tracker for AI answers. Everything above is input; there are only two honest feedback loops. 1. **Your CDN's AI bot analytics.** Cloudflare's AI Crawl Control reports actual AI bot request volume by operator. What you want to see is `GPTBot`, `ClaudeBot`, `OAI-SearchBot`, and `PerplexityBot` appearing with non-zero counts and no blocks. Give it two to four weeks. 2. **Ask the models directly, monthly.** Put the question your customer would actually type into ChatGPT, Claude, and Perplexity, and note whether you get cited. That is the outcome all of this aims at, and no dashboard reports it. --- ## The short version The uncomfortable truth about AI SEO is that most of it is just technical SEO done properly, plus one genuinely new requirement: **explicitly permitting AI crawlers, and describing yourself in a way a machine can resolve and verify.** If you only do three things: unblock the crawlers at your edge, build a real Schema.org entity graph with `@id` references and reciprocal `sameAs` links, and make your modification dates tell the truth. That is the 80%. Everything else on the list is refinement, and anything not on the list is probably someone's product. --- **Want a second opinion on your own setup?** I do this work on client sites — crawler policy, structured data, and the technical SEO underneath it. [Get in touch](/contact/) and I will tell you which of the six steps above you are actually missing. ## Frequently asked questions ### What does it mean to make a website AI-friendly? An AI-friendly website is one that AI crawlers are allowed to fetch, that states machine-readable facts about who published it, and that is structured so a model can extract a complete answer from a single section. In practice that means four things: not blocking AI bots at the CDN level, explicit per-bot rules in robots.txt, a Schema.org JSON-LD entity graph with stable identifiers, and an accurate sitemap with truthful modification dates. ### Is AI SEO different from regular SEO? Mostly no. AI answer engines lean heavily on conventional search indexes, so a site that is technically sound for Google is already most of the way there. The genuine additions are explicit crawler permissions for AI-specific bots, structured data that resolves you into an entity rather than a name string, and content shaped so a passage can be extracted and cited on its own. ### Do I need an llms.txt file? No. An Ahrefs study of 137,000 domains found that 97% of llms.txt files received zero requests in May 2026, Google stated in June 2026 that the file is not required for Search, and both OpenAI's and Anthropic's crawler documentation points site owners to robots.txt instead. It is worth adding only because AI coding agents such as Cursor and Claude Code do read it and the cost is one build step. Do not expect a citation lift from it. ### Do I need an auth.md file? Only if your service issues credentials. auth.md is an open protocol WorkOS published in May 2026 that tells an AI agent how to register on a user's behalf and receive a scoped, revocable token. Its real discovery path runs through OAuth Protected Resource Metadata and Authorization Server metadata, so it assumes you have an API behind a 401. A pure content site has no protected resource, no authorization server and no registration endpoint, which means publishing that discovery chain would advertise URLs that return 404. If you have nothing behind a login, the honest version is a short /auth.md stating that no registration exists and listing your public endpoints instead. ### Should I let AI crawlers train on my content? It depends on how you make money. If your goal is name recognition and inbound work, allowing training is usually the right trade, because being represented in a model's weights means being mentioned when someone asks a general question. If your content itself is the product, block the training crawlers and allow only the search and user-initiated ones. That keeps you citable in AI answers while opting out of training. ### How do I know if any of this is working? There is no rank tracker for AI answers. The two feedback loops worth watching are your CDN's AI bot analytics, which report real crawler request volume by operator, and asking ChatGPT, Claude, and Perplexity the questions your customers would ask, then noting whether you get cited. Give it two to four weeks before reading anything into either one.