Documentation · Markdown source · Edit on GitHub · CC0 1.0

VRP Spec — Test Coverage Analysis

Date: 2026-06-02 Scope: vacationrentalprotocol/vrp-spec (protocol specification, JSON Schema, JSON-LD context, example payloads, and the published website).

This document analyzes the current automated-verification coverage of the repository and proposes prioritized areas for improvement. It is an analysis and backlog, not a set of finished tests.

Follow-up status

The first validation gate now exists: npm test parses JSON/JSON-LD artifacts, validates Portable Attestations examples against schemas/attestations-v0.1.schema.json, and runs negative privacy/proof guardrails. This follow-up adds core schemas for discovery, JWKS, signed offers, and verifier results. Real Ed25519/JWS conformance vectors now also exist under examples/conformance/, verified end-to-end by npm test together with their failure modes (tampered payload, unknown kid, wrong key, expired valid_until).

The high-value gaps identified in this document now have automated coverage in npm test: core artifact schemas, real JWS offer vectors, real signed attestation-credential vectors verified against an issuer did:web document, three-state conformance, context/vocabulary consistency, status-list round trips, privacy negatives, and public route/link checks.

Baseline before validation PRs

Before the first validation PRs, the repository had no automated tests and no CI:

What ships in the repo that *is* mechanically checkable today:

Baseline observations from a one-off manual check (not yet automated):

Coverage gaps and proposed test areas

Priority 1 — Stand up a CI validation gate (foundational)

Everything below depends on having a place to run checks. Add a GitHub Actions workflow (and a thin package.json or a Python script) that runs on every PR and:

This converts the currently-implicit "the examples happen to be valid" into an enforced invariant. It is the single highest-leverage addition.

Priority 1 — Schema coverage for the *core* protocol artifacts

The JSON Schema only covers the Portable Attestations layer. The core VRP artifacts defined normatively in spec/v0.1.md §2–§5 have no machine-readable schema:

Proposed: author schemas/discovery-v0.1.schema.json, schemas/jwks-v0.1.schema.json, schemas/verified-stay-offer-v0.1.schema.json, and schemas/verified-stay-offer-verification-result-v0.1.schema.json, then validate the core examples against them in CI. Without these, the most important interop surface (what an agent actually fetches and verifies) is unguarded.

Priority 2 — Example/spec drift (already-present inconsistencies)

Tests would have caught these baseline issues; follow-up validation now covers the resolved structure:

  1. Two incompatible verified-stay-offer shapes. verified-stay-offer.not-quoteable.v0.1.json is the *signed-offer envelope* shape (kind, protocol_version, offer, signature). verified-stay-offer-verification-result.safe-to-quote.v0.1.json is a different *verification-result* shape (verified, agent_citation, official_offer_summary, agent_guardrails). These are now named as distinct object kinds and pinned by separate schemas.
  1. Signed payload missing SHOULD fields. The not-quoteable offer previously omitted request and property, which spec/v0.1.md lists in the signed payload. The example now includes them, and the signed-offer schema validates their shape when present. Since ADR 0012 the schema requires request, node_id and property, as §5 does; §7 compares request with the stay the agent asked for.

Priority 2 — Schema ↔ context ↔ vocabulary consistency (done)

contexts/v1.jsonld, schemas/attestations-v0.1.schema.json, and the repository copy of the /terms vocabulary page (terms.html) now stay in lockstep through CI validation:

Still useful later:

Priority 2 — Cryptographic / JWS test vectors (done)

Signing and verification (compact JWS, EdDSA over Ed25519) is the heart of the protocol. The placeholder examples elsewhere ("<compact JWS>", "<base64url-public-key>") are now complemented by real, verifiable vectors:

This gives independent implementers something concrete to test their verifiers against, which is the whole point of an interop spec.

Priority 3 — Behavioral / conformance vectors for agent rules (done)

spec/v0.1.md defines precise behavioral MUSTs (Safe-to-Quote §7, Fail-Closed §8, Three-State Verification §9) and examples/three-state- verification.md already works through four scenarios in prose. These are now covered by machine-readable fixtures plus one explicit fresh positive case:

npm test runs scripts/verify-three-state-fixtures.mjs against examples/conformance/three-state-verification.v0.1.json, keeping the prose examples honest and giving implementers a concrete behavioral conformance suite.

Priority 3 — Status list / revocation round-trip (done)

spec/attestations-v0.1.md §6 and examples/attestations/status-list.v0.1.json define revocation/suspension. npm test now checks that every example credential's credentialStatus resolves through statusListUrl / statusRef into the example status list, that issuer and purpose match, that credentialStatus.id is statusListUrl#statusRef, and that each referenced status yields a defined state (valid / revoked / suspended).

Negative checks cover missing status refs, issuer mismatch, purpose mismatch, duplicate status refs, and mismatched status-entry fragment IDs.

Priority 3 — Privacy guardrails as explicit negative tests (done)

The schema already forbids guest PII in VRPVerifiedStay subjects (not clauses for guestName, guestEmail, checkIn, guestScore, etc.). npm test now mutates the verified-stay example with every forbidden guest identity, exact-date, review, outcome, risk, score, and history field, and asserts that each payload fails schema validation.

Suggested sequencing

  1. Done: CI workflow + JSON parse + attestations-schema validation (P1).
  2. Done in follow-up: core-artifact schemas (discovery, JWKS, signed offer, verifier result) + validate examples (P1).
  3. Done: resolve the verified-stay-offer shape drift and add negative PII fixtures (P2).
  4. Done: real JWS test vectors + signature verification in CI (P2).
  5. Done: context/vocabulary consistency checks (P2-P3).
  6. Done: three-state conformance fixtures (P2-P3).
  7. Done: link checking and Vercel route checks (P4).
  8. Done: status-list round-trip checks (P3).

Steps 1–2 alone move the repo from "examples are validated by hand, sometimes" to "every PR proves the published artifacts are internally consistent," which is the core promise of an interoperability spec.