Introduction
At TestShift, we use our own site to work through two stages of agentic quality engineering. Each stage needs different evidence.
- Make the site readable to agents. We made the public content discoverable and retrievable, then tested the underlying HTTP and content contracts. Is Agentic recorded 100/100 for
test-shift.comon November 7, 2026. The first case study explains what we tested beyond the score. - Give agents actions they can invoke. We implemented three tools on TestShift’s contact page using WebMCP, an emerging browser protocol. Adopting it early gave us a chance to build automated Playwright tests into the integration itself, starting with two write actions: sending an inquiry and booking a consultation.
The score is a snapshot of stage one. It is not evidence that a write is safe. For engineering and QA leads, stage two raises a new release question: can we test what an agent is allowed to do, what the visitor approved, and what actually left the application? This is the hands-on follow-up to WebMCP for Governed AI and Deterministic Testing.
Expose actions, not vendor credentials
The contact page defines three tools through Chrome’s WebMCP JavaScript API. They’re available only when the browser supports the API; booking tools also need a configured backend.
| Tool | What it does | What the visitor approves |
|---|---|---|
send_contact_message | Sends to Formspree. | Reply address, topic and message. |
list_booking_times | Lists UTC slots; holds none. | Nothing to approve. |
book_consultation | Books through Calendly. | Invitee, time and time zone. |
The two write tools pause at an ordinary HTML dialog. The user confirms there; an input such as confirmed: true would not be consent. A browser hint may help the agent decide when to ask, but it’s not an enforcement mechanism. The booking server separately checks the challenge, session-bound intent, rate limits and slot availability. It consumes an intent once and does not automatically repeat a write when the provider’s response is lost. The ordinary form and Calendly link remain available.
The WebMCP draft makes registerTool() asynchronous. TestShift handles a rejected registration without breaking the ordinary contact page or preventing other tools from registering. Tool execution has a separate failure contract:
| Result | What the agent should do |
|---|---|
cancelled | Report that the operation did not proceed. It can mean the visitor declined or the page interrupted the operation; do not interpret it as provider rejection. |
requires_manual or verification_failed | Give the visitor the ordinary form or booking link and let them complete verification. Do not try the tool again automatically. |
unknown | Do not claim success or repeat the write. The provider may already have accepted it. Verify receipt with TestShift or check the Calendly invitation before trying again. |
Treat the setup as trust boundaries
Astro still builds the site as static pages. WebMCP runs in the browser on Contact; it doesn’t require an agent server or a rewrite of the site. The booking path adds a small set of Vercel Functions. Those functions, not the browser or agent, call the Calendly API with credentials. The contact path still submits to the existing Formspree form.
| Boundary | What we configured | What can go wrong | How we check it |
|---|---|---|---|
| reCAPTCHA and Formspree | Public site key at Astro build time; matching secret in Formspree; allowed hostname. | Wrong key pairing, token rejection, or a submission classified as spam. | A fixture checks all form fields, including a synthetic token when configured. A live check looks for the provider record, its classification and the email. |
| Booking flag and origin | An environment-scoped enable flag and the exact expected HTTPS site origin. | A copied Preview origin rejects valid public requests; enabling booking before configuration is complete exposes a broken path. | Configuration and origin checks in the handler, then a launch check against the deployed origin. |
| Turnstile and Redis intents | Hostname-scoped verification, a server-only signing key, shared short-lived state and rate limits. | A replayed intent, changed approved details, an expired challenge, or two processes accepting the same write. | Browser fixtures exercise the approval flow. Server tests check session/payload binding, one-time consumption and rejection paths. The in-memory E2E store does not validate Redis itself. |
| Calendly | A token with the required scopes, an event type selected by the server, and a meeting location compatible with that event. | Treating available times as reservations, using the wrong event, or booking without the required meeting location. | Fixture assertions inspect the complete provider payload. A controlled live check verifies the appointment, invitation and joining link separately. |
For production, provision separate scoped credentials, set the expected production origin, and validate the provider configuration for that origin. Keep secrets in the provider or in Vercel’s Secret environment variables. Configuration changes apply to new deployments; saving a value is not evidence that the deployment serving traffic uses it. Keep the booking flag off until the release checks pass. Test tool discovery in a supported browser without the Playwright testing flag as a separate launch check; consult Chrome’s current WebMCP availability guidance.
What is safe to publish? Describe the trust boundaries, the approval flow, and how the tests prove the contract. Use synthetic payloads and sanitized result codes. Leave out actual keys and tokens, resource and event IDs, private URLs, personal details, raw logs, and exact abuse-control settings. The server must enforce those controls even if an attacker knows the architecture.
Build the test environment before the assertions
Playwright 1.64 introduced page.webmcp and frame.webmcp. A test can now discover and invoke the same registered tool a browser agent would use, without involving an LLM or guessing which button represents the action. Then the same test can use ordinary Playwright locators to verify what the visitor sees and what they must approve. This gives QA a separate test surface: the agent-facing tool contract, the human approval UI, and the resulting provider request in one browser test.
TestShift runs this suite on Playwright 1.65.0 in a dedicated Chromium project with --enable-features=WebMCP. The regular browser tests do not need that flag.
A dedicated browser project
This is the relevant configuration, shortened from our Playwright config. The local server serves a built site; the fixture must proxy to the same port. The WebMCP API is experimental, so enabling it in a test is not evidence that every visitor’s browser exposes it.
import { defineConfig, devices } from "@playwright/test";
export default defineConfig({
testDir: "./e2e",
use: {
baseURL: "http://127.0.0.1:4321",
screenshot: "only-on-failure",
trace: "on-first-retry",
},
projects: [
{
name: "chromium",
testIgnore: "**/webmcp.spec.ts",
use: { ...devices["Desktop Chrome"] },
},
{
name: "webmcp-chromium",
testMatch: "**/webmcp.spec.ts",
use: {
...devices["Desktop Chrome"],
launchOptions: { args: ["--enable-features=WebMCP"] },
},
},
],
webServer: {
command: "pnpm preview --host 127.0.0.1 --port 4321",
url: "http://127.0.0.1:4321/",
reuseExistingServer: !process.env.CI,
},
});
Contact: intercept before navigation
The missing piece behind submissions is network interception. Inside the contact test, we install both routes before opening the page. The synthetic CAPTCHA implementation returns a known token; Formspree is never called. An attempted request is recorded even if a later assertion fails.
const submissions: {
method: string;
fields: [string, FormDataEntryValue][];
}[] = [];
await page.route("https://www.google.com/recaptcha/**", route =>
route.fulfill({
contentType: "text/javascript",
body: 'window.grecaptcha={ready(f){f()},execute(){return Promise.resolve("fixture-proof")}};',
})
);
await page.route("https://formspree.io/**", async route => {
const request = route.request();
const fields = await new Response(request.postDataBuffer()?.toString("utf8"), {
headers: { "content-type": request.headers()["content-type"] ?? "" },
}).formData();
submissions.push({
method: request.method(),
fields: [...fields].sort(([a], [b]) => a.localeCompare(b)),
});
await route.fulfill({ json: { next: "https://formspree.io/thanks" } });
});
The route controls the provider response, not the approval UI. Playwright still has to discover the real registered tool, open its real dialog and approve the displayed payload. The fixture token is not a CAPTCHA bypass for live use.
Booking: the real handler, controlled dependencies
The booking fixture starts a local Node HTTP server. It forwards page and asset requests to the static site and sends booking API requests to the real createBookingHandler. Only its external dependencies are replaced: Calendly responses, challenge verification and shared storage.
Here is the core, excerpted from that fixture. origin is the temporary local server’s origin and slot is a synthetic future UTC time; the HTTP adapter and page forwarding are omitted. Both factory functions are application code, not Playwright APIs.
const writes: unknown[] = [];
const values = new Map<string, string>();
const eventType = "https://api.calendly.com/event_types/fixture";
const calendly = createCalendlyAccess(
"fixture-token",
eventType,
async (input, init) => {
const url = new URL(String(input));
if (url.origin !== "https://api.calendly.com")
throw new Error("Unexpected fixture destination");
if (url.pathname === "/event_type_available_times")
return Response.json({
collection: [{ status: "available", start_time: slot }],
});
if (url.pathname !== "/invitees" || init?.method !== "POST")
throw new Error("Unexpected fixture operation");
writes.push(JSON.parse(String(init.body)));
return Response.json(
{ resource: { uri: "https://api.calendly.com/scheduled_events/fixture/invitees/test" } },
{ status: 201 }
);
}
);
const handler = createBookingHandler({
origin,
siteKey: "fixture-site-key",
hashKey: "fixture-only",
verify: async proof => proof === "fixture-proof",
...calendly,
store: {
async put(key, value) {
if (values.has(key)) return false;
values.set(key, value);
return true;
},
async take(key, value) {
if (values.get(key) !== value) return false;
return values.delete(key);
},
async limit() {
return true;
},
},
});
The browser’s request still travels through the application’s origin, cookie, intent and payload checks. writes records the final Calendly payload after those checks. This is broader than mocking the browser’s booking response. It is narrower than a deployed integration check: limit() always allows the request, the map has no Redis TTL or network behavior, and the synthetic proof does not exercise Cloudflare.
Test the tool contract and the human decision
Continue the contact test after installing the routes above. Start the tool call without awaiting it: it cannot return until the dialog is resolved.
await page.goto("/contact/");
await expect.poll(async () =>
(await page.webmcp.tools()).map(tool => tool.name)
).toContain("send_contact_message");
const pending = page.webmcp.callTool("send_contact_message", {
email: "test@example.com",
subject: "quality-gates",
message: "Our CI tests need a reliable signal.",
});
const review = page.getByRole("dialog");
await expect(review).toContainText("test@example.com");
expect(submissions).toHaveLength(0);
await review.getByRole("button", { name: "Send message" }).click();
expect(await pending).toMatchObject({ status: "accepted" });
expect(submissions).toHaveLength(1);
The suite checks three user journeys, plus two negative variants of the booking response:
| Scenario | What the browser test actually asserts |
|---|---|
| Send a contact message | Discover the tool, invoke it with synthetic data, and inspect the review. No provider request exists yet. A repeated page-load event must leave the pending review intact. After approval, parse the multipart request and compare the complete field set: reply address, topic, message, empty honeypot, and the fixture CAPTCHA token when configured. Require exactly one POST. Navigate away and verify that the contact tool is no longer registered. |
| Book a consultation | Open the Hebrew contact page. Discover both booking tools and read a known fixture slot. Verify the invitee address, time zone and exact UTC instant in the review. Require zero provider writes before approval; afterward, require booked, the localized success text, and exactly one provider payload with the expected event type, start time and invitee fields. |
| Cancel either write | Open both dialogs in turn, cancel, and assert that each tool returns cancelled and closes its dialog. The booking UI explains that no request was sent. Count attempted requests as well as recorded provider writes: no Formspree request and no POST to a booking write endpoint. |
| Reject misleading success signals | A booking result is shown as successful only for HTTP 201 with booked. HTTP 200 with booked, or HTTP 201 with unknown, must not show success. The UI instructs the visitor to check the invitation or calendar before trying again. |
The booking fixture exercises the real booking handler with controlled provider dependencies. The contact test intercepts the form request at the browser boundary. Neither suite uses a live account or proves that a CAPTCHA provider will accept a real token. Focused server tests cover a stale slot, a changed payload, replay of a booking intent, and a lost write response. If the provider might have committed a write, the safe result is unknown, not an automatic retry.
Make a misleading success fail the test
A successful HTTP status is not a confirmed booking. This adapted excerpt uses the local booking fixture above and a synthetic Turnstile implementation. It replaces only the booking response with HTTP 201 and unknown. The negative case must stay negative even though the transport says “created.”
test("201 with unknown must not show success", async ({ page }) => {
const fixture = await openBookingFixture();
try {
await page.route("https://challenges.cloudflare.com/turnstile/**", route =>
route.fulfill({
contentType: "text/javascript",
body: 'window.turnstile={render(_el,o){queueMicrotask(()=>o.callback("fixture-proof"));return "fixture";},remove(){}};',
})
);
await page.route("**/api/webmcp/book.json", route =>
route.fulfill({ status: 201, json: { status: "unknown" } })
);
await page.goto(fixture.origin + "/contact/");
await expect.poll(async () =>
(await page.webmcp.tools()).map(tool => tool.name)
).toContain("book_consultation");
const pending = page.webmcp.callTool("book_consultation", {
startTime: fixture.slot,
name: "Test Reader",
email: "test@example.com",
timeZone: "Asia/Jerusalem",
});
await page.getByRole("dialog")
.getByRole("button", { name: "Book consultation" }).click();
expect(await pending).toMatchObject({ status: "unknown" });
await expect(page.getByLabel("Booking status")).toHaveText(
"Calendly did not confirm the result. Check your invitation or calendar before trying again."
);
} finally {
await fixture.close();
}
});
The companion case returns HTTP 200 with booked. Together they enforce the conjunction: only 201 and booked produce the success UI. They do not exercise a live Calendly failure; they test how our client interprets the response.
Reproduce the race, then keep it fixed
A live Preview invocation returned cancelled after the visitor approved it. No request reached Formspree. The useful diagnostic was the sequence, not the final string:
- The contact module initialized and registered its tools before the page finished loading.
- The visitor approved a tool invocation; it started obtaining verification proof.
- A late
astro:page-loadevent ran setup again. - Setup called cleanup, which aborted the active invocation. Formspree never received a POST.
To recreate that ordering, we kept the contact interception above, delayed a synthetic image to hold page loading open, and delayed the fixture CAPTCHA proof for longer. This is a diagnostic fragment, not a standalone test: continue with the same tool invocation and approval sequence after the navigation. The delays are deliberate fault injection, not sleeps used to decide when a real page is ready.
await page.route("**/__webmcp_fixture_delay.gif", async route => {
await new Promise(resolve => setTimeout(resolve, 1400));
await route.fulfill({ status: 404, body: "" });
});
await page.addInitScript(() => {
document.addEventListener("DOMContentLoaded", () => {
const blocker = new Image();
blocker.src = "/__webmcp_fixture_delay.gif";
document.body.append(blocker);
});
});
await page.route("https://www.google.com/recaptcha/**", route =>
route.fulfill({
contentType: "text/javascript",
body: 'window.grecaptcha={ready(f){f()},execute(){return new Promise(r=>setTimeout(()=>r("fixture-proof"),2200))}};',
})
);
await page.goto("/contact/", { waitUntil: "domcontentloaded" });
Waiting for the default full load in that reproduction would miss the vulnerable window. Starting at domcontentloaded let us approve while loading and verification were still in flight. The recording showed the late page-load event aborting the operation. With the fix in place, the same controlled timing completed with one intercepted POST and accepted.
Initialize once per page, dispose on navigation
The mistake was treating every setup event as permission to tear down the existing tools. This is the lifecycle block from Contact; initContactForm, initBookingCalendar and registerContactTools are application helpers imported by that module. The guard prevents the second initialization from cancelling in-flight work; navigation still disposes the old page’s tools.
let cleanup: (() => void) | undefined;
function setupContact() {
if (cleanup) return;
const root = document.querySelector<HTMLElement>("[data-contact-form]");
const widget = document.getElementById("calendly-widget");
if (!root && !widget) return;
const cleanForm = root ? initContactForm(root) : () => {};
const cleanBooking = widget ? initBookingCalendar(widget) : () => {};
const form = root?.querySelector("form");
const dialog = document.querySelector<HTMLDialogElement>("[data-webmcp-review]");
const cleanTools = form && dialog
? registerContactTools({
document, form, dialog,
locale: document.documentElement.lang === "he" ? "he" : "en",
})
: () => {};
cleanup = () => {
cleanTools();
cleanForm();
cleanBooking();
};
}
document.addEventListener("astro:page-load", setupContact);
document.addEventListener("astro:before-swap", () => {
cleanup?.();
cleanup = undefined;
});
setupContact();
Keep the regression deterministic
The checked-in contact test does not depend on the diagnostic delay values. It dispatches a repeated page-load event while the review is open, checks that the review survives, then approves and requires one intercepted write:
const pending = page.webmcp.callTool("send_contact_message", contact);
const review = page.getByRole("dialog");
await expect(review).toContainText(contact.message);
expect(submissions).toHaveLength(0);
await page.evaluate(() => document.dispatchEvent(new Event("astro:page-load")));
await expect(review).toBeVisible();
await review.getByRole("button", { name: "Send message" }).click();
expect(await pending).toMatchObject({ status: "accepted" });
expect(submissions).toHaveLength(1);
This regression checks repeated setup during review. The delayed-proof reproduction covers the related post-approval timing. Neither justifies suppressing real cancellation or navigation. Before submission, both must still stop the work; after a write leaves the browser, cancellation cannot guarantee that the provider did not commit it.
A passing test and a confirmed booking are different evidence
The automated suite checks the browser tool, the approval UI and the request at the provider boundary. It uses provider fixtures, so it can run repeatedly without sending mail or taking calendar slots. It does not prove that a live provider will accept the request in a particular environment.
In controlled Preview checks, the availability tool returned bookable slots from the live provider. These are appointment times, not the host’s working-hours settings or a reservation. We then verified one approved booking through WebMCP: an active appointment at the provider and a confirmation email with a Zoom joining link.
After the lifecycle fix, one live contact attempt in Preview made exactly one POST. Formspree returned HTTP 200, and the matching submission appeared in its Spam folder. No matching notification was found in the destination mailbox. That proves the request reached the provider; it does not prove inbox placement or email delivery. We did not infer which spam signal caused the classification.
After rollout, I checked the flows manually on the production site as a separate release check. That is evidence about the deployed setup at the time of the check, not an assertion made by the fixture suite or a guarantee about later deliveries.
The release lesson: define success where the write is confirmed, not where the tool stops running. Chrome DevTools reports the invocation status separately from the tool’s output. A call marked Completed can still return a business failure. Show the outcome in the application, and use provider-side evidence to verify a controlled live write. If the outcome is unknown, check for the resulting record before another attempt.
Key takeaways
- Separate readability from permission to act. An audit can check whether agents can reach and read content. Treat every exposed write as a separate release decision.
- Test the tool and the approval together. For a first rollout, start with one bounded action. Require zero writes before approval, one observed request with the exact approved payload afterward, and zero writes on cancellation.
- A lost response is not a failed write. The provider may already have committed it. Return an unknown result with a concrete way to verify what happened, not an automatic retry.
- Keep the evidence in scope. For contact, verify provider receipt, classification and email separately. For booking, check the appointment and the invitation. A green fixture does not prove production is configured correctly, and a live write does not prove that a model will select the right tool.
- Test the gap between approval and the write. When the dialog closes but no POST leaves, inspect the lifecycle before blaming the provider. In our case, a second initialization was enough to cancel an approved action.