Verifier

What polyxd-verify checks, how to run it, how the v0 score works, and what it has found so far.

@polyxd/verifier scores a UI document the way it will actually be used: rendered, in every design system, by people and by agents. It is the check that runs before a generated interface ships, whichever generator wrote it, and it is how you compare generators on equal terms.

What it checks#

A document that fails the spec schema or structural rules is not rendered.

Document checks in detail#

Besides the validator, pattern, capability and rule checks, the document layer adds agent-readiness heuristics:

Rendered checks in detail#

Other renderers#

The rendered checks run through a harness page that renders window.__PXD__ and records what the surface sends; the React and Web Components renderers each ship one, and any renderer can provide its own (verifyDocument(doc, { harness: { url } | { html } })). npm run conformance -w @polyxd/verifier holds the two shipped renderers to the same DOM, ARIA, text and findings across every document and pack. See Renderers.

The default matrix#

Material 3, Carbon and Ant Design × light and dark × 390px and 1100px wide. That is 12 renders per document.

CLI#

npm install -D @polyxd/verifier playwright
npx playwright install chromium   # once

# one or more documents
npx polyxd-verify my-ui.json other-ui.json

# a narrower matrix, with a capability registry, agent tasks and a JSON report
npx polyxd-verify my-ui.json \
  --themes carbon,antd --modes light --widths 390 \
  --registry capabilities.json \
  --tasks tasks.json \
  --json report.json

Inside the Polyxd repository, npm run verify:examples -w @polyxd/verifier runs every spec example through the full matrix with agent tasks.

Output looks like this:

100  money-send-confirm  (0 errors, 0 warnings agent 12/12)

1 documents · 12 renders · mean score 100.0 · agent tasks 12/12

Without --quiet, each distinct finding is listed once with the check id, message and the first target it appeared in. The CLI exits with 1 if any document has an error or a failed agent run.

From code#

import { verifyDocument, compare } from "@polyxd/verifier";

const report = await verifyDocument(doc, {
  registry,                     // capability registry
  rules: direction.rules,       // Design Direction or journey acceptance rules
  tasks,                        // agent tasks for this document
  themes: ["material3"], modes: ["light"], widths: [390],
});

report.score;        // 0–100
report.static;       // document findings
report.targets;      // per theme/mode/width: findings and agent results

Playwright is an optional peer dependency. The rendered checks need it, so install it beside the verifier, as above. Nothing loads it until a browser is launched. Without it, launch() says what to install.

Document checks only#

@polyxd/verifier/static has the document checks on their own. It imports no Playwright and reads no files, because the spec's pattern checks are built into it. So it runs in Node, browsers and Workers, and you don't need Playwright to use it.

import { staticAudit } from "@polyxd/verifier/static";

const findings = staticAudit(doc, {
  registry,                  // optional capability registry
  rules: direction.rules,    // optional Design Direction or acceptance rules
  emphasisBudget: 1,         // optional: primary actions allowed in one view
  missingData: "warning",    // optional: how a binding that reads nothing is reported. Default "error"
});

The runtime runs these checks on every answer by default. The MCP server and the generation server use them for their verify tools.

Score (v0)#

Start at 100. Then:

• Each distinct failing error check costs 20. A check that fails in all 12 renders still costs 20 once, so one broken thing isn't counted twelve times.
• Each distinct warning costs 4.
• Failed agent runs cost up to 40, in proportion: 40 × (1 − successes / runs).
• The score floors at 0. A document that fails the spec schema or structure scores 0.

These weights are a starting point. npm run gold -w @polyxd/verifier measures them against a designer's ranking. Known blind spot in v0: vaguer labels, terser empty-state copy and reordered actions aren't checked, so two of the mild-drift variants score the same as their originals.

What the score is not#

A designer has ranked generated output: twelve requests, three generated options each, best to worst, with notes (bench/rank-set/ranking.json, npm run gold -w @polyxd/verifier -- --model). The verifier agrees with none of it — 0% exact order, mean Kendall tau-b −0.29.

That number is the most useful result the verifier has produced, because of how it disagrees:

• In six of the twelve groups it can't separate the options at all. All three score the same. Whatever the designer saw — the amount not being the biggest thing on a payment screen, an information architecture that reads in the wrong order, a horizontal bar that means nothing — the score is blind to it.
• Where it does discriminate, it mostly runs backwards. Five groups score −0.82 or −1.00.
• It rewards a document for being small. On "stop emailing me", the verifier's favourite has one toggle and the designer's has four: the one-toggle version can't trip the ambiguous-name check, has fewer inputs and doesn't attempt the job. The designer ranked it last, because "stop emailing me" is a question about which emails.
• A document that doesn't attempt the task can't make mistakes. On "find 30 minutes with Tom and Priya", the two options that tried to send the invitation now carry an error for doing it from a card click; the option that does nothing at all scores 96 and comes top. The designer ranked that one last.

None of this means the checks are wrong — the three added from this ranking each catch something real (see below). It means the score is a floor, not a ranking: it answers is this correct, accessible and operable by an agent, and it has no term at all for does this do the job it was asked to do. Those are two numbers, and flattening them into one makes both worse.

It measures damage, not quality#

The same designer also ranked the hand-made gold set — thirty documents in ten groups of original, mild drift and clearly worse. There the verifier agrees: 50% exact order, mean tau-b +0.63.

So the verifier is reliable at "is this a degraded version of a good interface" and has nothing to say about "which of these three honest attempts is best". That second case is the one that matters when a generator produces several candidates and something has to pick one.

What the reward does about it#

The reward, not the score, is what picks between candidates when a generator produces several (--reward measures that instead). It reads what the bench already declares per request — the capability an answer has to wire, and the control the task presses by name — as a coverage term, and coverage multiplies rather than adds:

reward = score × (0.2 + 0.8 × coverage) + 20 if an agent could act

Multiplying matters. On "find 30 minutes with Tom and Priya", the candidate that attempted nothing scored 96 and the two that tried to send the invitation scored 56, because trying earned them two errors. Adding a coverage bonus left the empty one ahead; multiplying puts it last, which is where the designer put it.

Three honest numbers about this, measured rather than hoped for:

Coverage separates the candidates in 1 of 12 groups. In ten of the twelve, all three candidates cover 100% of what the request asked for and the term is silent. So coverage is a gate, not a ranker: it catches the surface that attempted nothing, a real failure mode that picking by score alone would reward, and adds no ordering signal beyond that. Nothing computable that we have separates honest attempts, which is why taste has to come from a designer's preferences rather than from a check.

The length penalty is gone. It existed so that padding a document to satisfy checks wouldn't pay, but it pushed the same way as everything else: on "stop emailing me" the designer ranked the four-toggle version first and the one-toggle version last, and the reward was already biased towards the small one.

A model judge agrees with other models, not with the designer#

If the verifier can't rank honest attempts, can a frontier model? The experiment: 23 groups from both ranking rounds, each candidate rendered at 390 and 1100 px, option letters shuffled per group, judged blind by agents with no access to the designer's ranking — the same screenshots the designer saw. Then the whole thing again with a second, independent judge as a control.

The control is the result. Two judges who never met agree strongly with each other and with the designer at chance — on the 15 groups where they ranked identically, agreement with the designer is still only 0.24.

So the task is reliably judgeable, and models converge on an answer. It is simply a different answer. Read the judges' reasons and the split is plain: they rank on defects in the artefact — a binding that renders [object Object], an empty state where data exists, a missing call to action, copy that repeats itself. The designer ranked on which thing should be biggest, what order the page reads in, whether the visual form suits the data, and what the equivalent screen looks like in products they know.

This is the project's premise, measured rather than asserted: conventional quality is recoverable from the artefact; taste is not. A model judge is worth having as a second verifier — it found three real defect classes the static checks missed, and they are checks now — but it cannot stand in for a designer, and neither a faster nor a cheaper judge would change that, because speed was never what was missing.

The designer agrees with their own re-rank as much as the models agree with each other#

The control on the other side: six already-ranked groups, shown again under freshly shuffled letters. npm run rerank -w @polyxd/verifier builds it, -- --score compares the two sittings.

Two reliable raters, each reproducing itself, measuring different things. That settles it: the rankings are not noise, and the disagreement with models is not a failure of either side — it is the gap the project exists to cross.

Where the rater flips, the options are equivalent. The two groups the designer re-ordered are the two where independent judges also called the candidates near-identical ("near-identical confirm dialogs", "2 and 3 are clean"). That is information, not error: it says those pairs carry no preference. The numbers follow — the best-versus-worst pair survived in 5 of 6 groups, the adjacent pairs in 10 of 12. So anyone using these rankings as preference data should weight a pair by the distance between its members and drop the ones a re-rank reverses, rather than treating all three pairs in a group as equal evidence.

The three checks this ranking added#

The three the model judges added#

The two the approved designs added#

Rating one generator's output against the approved flow designs, 8 of 12 screens showed nothing real: the layout was plausible and every value was blank. A binding to a path that isn't in the data had been a warning, costing 4 points, so a screen of blanks could still score in the 90s.

The first blind review against the designs named three more, now checks too:

The same change fixed how the verifier reads bindings inside a repeated item. It now resolves them the way the renderer does: a relative path from the item, an absolute path from the top of the data wherever it appears.

Consistency#

compare(previous, current) measures whether something the user saw last time still looks and sits the same way. It matches the two documents by semantic key, not by id.

const { score, parts, differences } = compare(lastTime, now);
// differences: ['"fee" changed from DetailList.item to Text', '"reference" relabelled: "Reference" → "Note"']

The score runs from 0 (nothing recognisable) to 1 (same structure, order and labels). The benchmark's multi-turn sequences use it to measure consistency across turns.

Results today#

• Injected defects: 20 of 20 deliberately injected defects are caught (test/defects.test.ts). They span schema, structure, patterns, capabilities, copy, rendered accessibility, layout and agent operability: two primary actions, a destructive capability outside a confirmation, a generic "OK" confirm label, results before filters, eight inputs in one view, an image without alt text, a hard-coded balance, a required field removed so the task can't be done, unbreakable text overflowing on a phone, and more. The Phase 3 exit test asks for at least 90%.
• Examples: all 24 spec examples score 100 across 1,248 renders — 13 design-system packs at two widths, both modes.
• Agents: 216 of 216 agent task runs succeed (18 tasks × 12 targets).

Bugs it found in the renderer#

Building the verifier found real bugs in @polyxd/react, all since fixed:

• Graphics-only colours were used for text, which failed WCAG contrast in Carbon and Ant Design.
• Selection rows were smaller than each pack's target size.
• "(required)" markers and option descriptions leaked into accessible names.
• Root confirmations blocked the host page.

Polyxd is an early preview. Found something unclear? It will get better with your feedback.