How to Test a Website for AI Agents: Beyond the Is Agentic Score

An AI agent does not experience a website the way a person does.

It may never paint the page. It may not execute JavaScript. It may request Markdown, follow a sitemap, inspect structured data, or decide that a path exists based only on an HTTP status code. A site can look excellent in Chrome and still be ambiguous, expensive, or actively misleading to an agent.

That creates a new quality problem:

How do we test whether a public website is genuinely usable by agents?

Not “Does ChatGPT know the brand?” Not “Did a scanner return a high number?” And not “Did we add an llms.txt file?”

The engineering question is whether the site exposes a coherent, testable contract across discovery, retrieval, semantics, recovery, identity, and production routing.

This is the discipline TestShift is building: agent-readiness quality engineering. The work began with a public audit and produced a deterministic test architecture rather than a collection of SEO decorations.

The first implementation moved TestShift from 76 to 91. The latest Is Agentic audit reports 97/100. More important than the score, the work exposed protocol bugs that a visual test would never find.

What Agent-Ready Actually Means

An agent-ready site makes its public knowledge easy to discover, retrieve, understand, and verify without pretending that every crawler behaves the same way.

For TestShift, that contract has six layers:

Agent readiness is not a replacement for accessibility, security, SEO, or API quality. It intersects with all four, but it has its own failure modes.

A screen-reader audit will not necessarily catch a wrong Content-Type. A visual regression will not tell you that a nonexistent URL looks real to a crawler. An SEO tool may reward schema that exposes data the business should keep private.

That is why this needs a quality architecture, not a checklist copied into production.

The New Signal: Is Agentic, Built by Vercel on Ora

Is Agentic is a public readiness scanner built by Vercel on the Ora audit engine. Its methodology separates Essential and Recommended checks, adds a capped bonus, and describes every report as a point-in-time snapshot rather than a certification.

That framing matters.

A useful audit is a probe. It gives us evidence, hypotheses, and a reproducible starting point. It does not become product truth simply because the output is numerical.

The first TestShift scan scored 76/100. Follow-up production fixes raised the August 26, 2026 scan to 97/100. The remaining findings were:

1. Content without JavaScript: partial.
2. Brand-name discoverability: partial.

Those findings required different QA responses.

• The 404 evidence exposed a real defect and justified code; it now passes.
• The no-JavaScript evidence required investigation and a targeted semantic improvement, not markup vandalism.
• Brand ranking depended partly on external indexing and required honest long-term work, not invented business data.

This distinction is the heart of senior quality engineering:

A finding is evidence to investigate, not permission to change the system blindly.

Architecture: One Page, Two Static Representations

TestShift is an Astro site. Every page is statically generated.

The agent-ready architecture keeps that property. It does not introduce runtime page rendering and it does not ask an LLM to convert content on demand.

The build pipeline now runs in this order:

Astro static generation
  -> HTML-to-Markdown sibling generation
  -> Pagefind indexing
  -> Pagefind asset sync
  -> built SEO audit
  -> built agent-readiness audit

For every page route, the build produces two representations:

/services/quality-gates/index.html
/services/quality-gates/index.md

The canonical URL remains /services/quality-gates/. A browser receives HTML. A client that explicitly sends Accept: text/markdown receives the prebuilt Markdown sibling.

The Markdown contains the canonical URL and the meaningful content from main#main-content. Presentation-only navigation, scripts, styles, buttons, desktop table-of-contents chrome, and hidden elements are removed. Headings, paragraphs, links, images, lists, blockquotes, code blocks, and tables survive.

At the original release gate, the build verified 145 HTML pages and 145 Markdown siblings.

This is not a parallel content system. It is one source rendered into two deterministic artifacts.

The Routing Bug That Looked Correct

The first implementation used conditional rewrites and headers in vercel.json.

On paper, it looked reasonable:

Accept: text/markdown
  -> rewrite /page/ to /page/index.md
  -> set Content-Type: text/markdown

In production, Vercel resolved the existing static HTML page before that rewrite phase. The response body was HTML, while the conditional header said Markdown.

That is worse than no Markdown support. It is a false protocol promise.

The fix moved negotiation to root Vercel Routing Middleware, which runs before the static-filesystem lookup. The middleware is deliberately narrow:

• only GET and HEAD;
• only root or trailing-slash page routes;
• only an enabled text/markdown media range;
• never RSS, sitemap XML, OG images, assets, or other file-like routes;
• q=0 means Markdown is not acceptable;
• browser requests continue normally.

Correct negotiated responses include:

Content-Type: text/markdown; charset=utf-8
Vary: Accept, Accept-Encoding

Direct index.md artifacts also return X-Robots-Tag: noindex, so the canonical HTML URL remains the indexable address.

The lesson is simple:

Never verify content negotiation by looking at headers alone. Assert status, media type, and body together.

The 404 Bug Behind the Earlier 91

The 91/100 report said the site returned a real 404 but lacked a short Markdown recovery body.

The HTML 404 already contained localized links to the homepage, insights, llms.txt, and sitemap. So why did the scanner still see only partial support?

The production probe revealed a deeper bug.

A request such as:

curl -i -H "Accept: text/markdown" \
  https://www.test-shift.com/agent-readiness-does-not-exist/

was rewritten toward a Markdown sibling that did not exist. Vercel correctly fell back to the HTML 404 and kept status 404, but a canonical-route header still labeled that HTML body as text/markdown.

Real status. Wrong representation.

The follow-up fix does three things:

1. The middleware probes the prebuilt Markdown sibling before rewriting.
2. If it is missing, it serves the generated 404.md recovery artifact with status 404.
3. It sets Content-Type, Vary, and X-Robots-Tag on that response while removing the unsafe canonical content-type override.

The generated Markdown 404 now gives an agent compact next steps:

• homepage;
• insights index;
• llms.txt agent guide;
• XML sitemap.

It is also derived from the visible 404 source, not maintained as a disconnected second copy.

That is what an agent-friendly error means: truthful failure plus a recoverable path forward.

Content Without JavaScript: Test the Evidence, Not the Label

The latest scanner report found 3,348 characters and an H1, but still described the homepage heading structure as flat.

Those statements do not fully agree.

The raw production HTML is already statically rendered. The deterministic audit found meaningful copy, exactly one H1, multiple H2 and H3 headings, and no skipped heading levels. The hero summary is now a visible branded H2, preserving its typography while giving shallow extractors an immediate H1-to-H2 relationship. Playwright also verifies the localized H1 and hero H2 in the raw English and Hebrew documents.

Changing the visual hierarchy merely to satisfy a contradictory scanner label would damage valid semantics.

The correct QA response is:

1. reproduce with raw HTML, not a hydrated DOM;
2. count content characters inside the canonical main region;
3. assert one H1;
4. assert useful lower-level headings;
5. reject skipped levels such as H1 directly to H3;
6. make the smallest semantic correction when the raw evidence identifies an unambiguous gap;
7. recheck after deployment and cache expiry.

This is why TestShift treats external scanners as test oracles with known limitations. They can find risks. They can also be stale, opaque, or wrong.

Do not game a scanner. Build stronger evidence than the scanner.

Brand Discoverability Is Not a Schema Fabrication Contest

The remaining Recommended finding placed test-shift.com sixth among nine brand-search results for “TestShift.”

Some work belongs in the repository:

• one canonical https://www.test-shift.com identity;
• consistent TestShift and TestShift by Nir Tal names;
• aligned Organization, WebSite, founder, and social-profile signals;
• locale-aware canonical and alternate URLs;
• no redirect chains hiding the canonical domain;
• first-party articles that connect TestShift to agent-readiness quality engineering.

Some work does not belong in code:

• earning independent mentions and links;
• maintaining consistent business listings;
• claiming profiles that require owner credentials;
• waiting for search engines to recrawl and rerank the brand.

The audit suggested stronger NAP data. TestShift deliberately added only a truthful country-level address: Israel. It did not invent a street, city, postal code, or telephone number for points.

Privacy and truth outrank scanner completeness.

This article is part of the legitimate brand strategy: publish concrete engineering evidence, make it citable, and connect the TestShift name to a repeatable discipline rather than to a slogan.

The Test Architecture Behind the Change

The important outcome was not llms.txt. It was the regression system around the whole contract.

At the latest verification, the deterministic suite reported 23 Vitest files, 140 passing tests, and 2 explicit TODOs. The numbers will evolve. The layers are what matter.

1. HTML-to-Markdown Unit Tests

The converter tests prove that machine-readable content retains:

• canonical URLs;
• headings and paragraphs;
• links, lists, and images;
• blockquotes;
• inline and fenced code;
• tables;
• English and Hebrew text.

They also prove that scripts, styles, hidden content, presentation chrome, and desktop-only table-of-contents elements do not leak into the artifact.

The 404 follow-up added another important rule: recovery navigation may opt into agent content while ordinary navigation stays excluded.

2. Build-Generation Tests

The generator walks built page artifacts and creates a Markdown sibling for every index.html and the special 404.html.

Tests prove that it:

• generates every expected sibling;
• includes the 404 artifact;
• ignores unrelated HTML files;
• fails loudly and names the malformed source page when main#main-content is missing.

A silent partial build would be more dangerous than a failed build, so incompleteness is fatal.

3. Routing-Middleware Unit Tests

The middleware suite covers the protocol decision table:

The test asserts the body and recovery links, not only the rewrite header.

4. Built-Artifact Contract Tests

pnpm check:agent audits the final static output after the build.

It rejects:

• a page without a Markdown sibling;
• an empty sibling or one that lost the primary heading;
• missing representative English or Hebrew routes;
• weak raw homepage content or invalid heading order;
• incomplete HTML or Markdown 404 recovery;
• incomplete positive and negative guidance in llms.txt;
• unsafe private Organization fields;
• inconsistent Organization and WebSite identity;
• high-level Markdown rewrites in vercel.json;
• canonical content-type overrides that can mislabel HTML 404s;
• incomplete direct-artifact headers or Routing Middleware.

This turns agent readiness into a release invariant.

5. Playwright Browser Tests

Playwright verifies the user-visible side of the same contract:

• English and Hebrew pages render with the correct locale and direction;
• raw localized homepage content exists before client behavior matters;
• the 404 remains a real HTTP 404;
• English and Hebrew recovery links are correct;
• direct 404.md contains the agent recovery routes;
• representative articles expose matching HTML and Markdown;
• the skip link still targets main#main-content;
• route and accessibility behavior remain intact.

Astro Preview cannot execute Vercel Routing Middleware or deployment headers. That boundary is explicit. Playwright validates the built site; live HTTP probes validate the platform behavior.

6. Production Contract Tests

The final gate checks deployed endpoints with curl:

# Browser representation
curl -I -H "Accept: text/html" https://www.test-shift.com/

# Negotiated representation
curl -I -H "Accept: text/markdown" https://www.test-shift.com/

# Direct artifact
curl -I https://www.test-shift.com/index.md

# Agent-friendly missing route
curl -i -H "Accept: text/markdown" \
  https://www.test-shift.com/this-route-must-not-exist/

# File-like route must stay XML
curl -I -H "Accept: text/markdown" https://www.test-shift.com/rss.xml

For each response, assert status, Content-Type, Vary, indexing policy, and a body marker. A 200 header test alone is not enough.

A Practical Agent-Readiness Quality Gate

If I were introducing this discipline into an engineering organization, I would use this order.

Gate 1: Truthful HTTP

• nonexistent routes return 404 or 410;
• HTML is labeled HTML;
• Markdown is labeled Markdown;
• negotiated routes vary on Accept;
• file-like endpoints remain unnegotiated.

Gate 2: Deterministic Content

• useful content exists in raw HTML;
• one clear H1 and a valid heading hierarchy;
• every canonical page has a generated machine-readable representation;
• conversion is loss-aware and fails closed.

Gate 3: Recovery and Discovery

• sitemap and llms.txt are reachable;
• 404 responses provide compact next steps;
• agent guidance says when the organization is relevant and when it is not.

Gate 4: Identity Without Fiction

• canonical domain, Organization, WebSite, founder, and visible copy agree;
• locale alternates remain aligned;
• no private or invented fields are published for a score.

Gate 5: Platform Reality

• unit tests verify decision logic;
• built audits verify generated artifacts;
• Playwright verifies browser and locale behavior;
• Preview verifies hosting semantics;
• production probes verify the public contract;
• external scans provide an independent signal, never final authority.

This is a familiar TestShift pattern: probabilistic discovery on the outside, deterministic evidence at the gate.

The Strategic Shift: Agents Are a New Consumer of Your Platform

Most organizations still treat agent access as a content task. Add a file. Add schema. Ask for another crawl.

That is too small.

Agents are becoming a new class of platform consumer. They have transport preferences, context limits, recovery needs, identity questions, and failure modes. The interface may be HTML, Markdown, an API, or WebMCP, but the quality problem is the same: does the system expose a truthful, governed contract?

TestShift is bringing the methods of modern test architecture into that space:

• model the consumer;
• define the contract;
• isolate deterministic and probabilistic behavior;
• test the build artifact;
• test the deployment platform;
• preserve human experience;
• publish evidence rather than claims.

The score is useful. The score is not the product.

The product is trust.

And this is the space TestShift intends to lead: testing and governing how software systems become usable by agents without becoming less truthful for humans.

If your platform team is preparing a public website, developer portal, test platform, or internal knowledge surface for AI agents, TestShift Quality Gates can turn that goal into an executable release contract. Contact TestShift to design the audit, protocol tests, Playwright coverage, and production verification around your architecture.

Architecture > Magic. Evidence > Score.