Verifier

What polyxd-verify checks, how to run it, how the v0 score works, and what it has found so far.

@polyxd/verifier scores a UI document the way it will actually be used: rendered, in every design system, by people and by agents. It is the check that runs before a generated interface ships, and it will be the reward signal when the model is trained.

What it checks#

A document that fails the spec schema or structural rules is not rendered.

Document checks in detail#

Besides the validator, pattern, capability and rule checks, the document layer adds agent-readiness heuristics:

Rendered checks in detail#

The default matrix#

Material 3, Carbon and Ant Design × light and dark × 390px and 1100px wide. That is 12 renders per document.

CLI#

Inside the monorepo (the packages are not on npm yet; they are coming with v0.1):

# once: build the render harness the verifier loads in Chromium
npm run build:harness -w @polyxd/verifier

# one or more documents
npm run verify -w @polyxd/verifier -- my-ui.json other-ui.json

# a narrower matrix, with a capability registry, agent tasks and a JSON report
npm run verify -w @polyxd/verifier -- my-ui.json \
  --themes carbon,antd --modes light --widths 390 \
  --registry capabilities.json \
  --tasks tasks.json \
  --json report.json

# all 20 spec examples, full matrix, with agent tasks
npm run verify:examples -w @polyxd/verifier

When published, the binary is polyxd-verify. Output looks like this:

100  money-send-confirm  (0 errors, 0 warnings agent 12/12)

1 documents · 12 renders · mean score 100.0 · agent tasks 12/12

Without --quiet, each distinct finding is listed once with the check id, message and the first target it appeared in. The CLI exits with 1 if any document has an error or a failed agent run.

From code#

import { verifyDocument, compare } from "@polyxd/verifier";

const report = await verifyDocument(doc, {
  registry,                     // capability registry
  rules: direction.rules,       // Design Direction or journey acceptance rules
  tasks,                        // agent tasks for this document
  themes: ["material3"], modes: ["light"], widths: [390],
});

report.score;        // 0–100
report.static;       // document findings
report.targets;      // per theme/mode/width: findings and agent results

Score (v0)#

Start at 100. Then:

• Each distinct failing error check costs 20. A check that fails in all 12 renders still costs 20 once, so one broken thing isn't counted twelve times.
• Each distinct warning costs 4.
• Failed agent runs cost up to 40, in proportion: 40 × (1 − successes / runs).
• The score floors at 0. A document that fails the spec schema or structure scores 0.

These weights are a starting point. The gold set in bench/gold (30 documents in 10 groups of original, mild drift and clearly worse) exists to calibrate them against a designer's ranking with npm run gold -w @polyxd/verifier. The human ranking has not been recorded yet. Known blind spot in v0: vaguer labels, terser empty-state copy and reordered actions aren't checked, so two of the mild-drift variants score the same as their originals.

Consistency#

compare(previous, current) measures whether something the user saw last time still looks and sits the same way. It matches the two documents by semantic key, not by id.

const { score, parts, differences } = compare(lastTime, now);
// differences: ['"fee" changed from DetailList.item to Text', '"reference" relabelled: "Reference" → "Note"']

The score runs from 0 (nothing recognisable) to 1 (same structure, order and labels). The benchmark's multi-turn sequences use it to measure consistency across turns.

Results today#

• Injected defects: 20 of 20 deliberately injected defects are caught (test/defects.test.ts). They span schema, structure, patterns, capabilities, copy, rendered accessibility, layout and agent operability: two primary actions, a destructive capability outside a confirmation, a generic "OK" confirm label, results before filters, eight inputs in one view, an image without alt text, a hard-coded balance, a required field removed so the task can't be done, unbreakable text overflowing on a phone, and more. The Phase 3 exit test asks for at least 90%.
• Examples: all 20 spec examples score 100 across 240 renders.
• Agents: 216 of 216 agent task runs succeed (18 tasks × 12 targets).

Bugs it found in the renderer#

Building the verifier found real bugs in @polyxd/react, all since fixed:

• Graphics-only colours were used for text, which failed WCAG contrast in Carbon and Ant Design.
• Selection rows were smaller than each pack's target size.
• "(required)" markers and option descriptions leaked into accessible names.
• Root confirmations blocked the host page.

Polyxd is an early preview. Found something unclear? It will get better with your feedback.