Introduction

Veredictum is a conformance instrument for openEHR clinical data repositories. You point it at a running CDR and it tells you, with citations, which parts of the openEHR specification that server implements.

It executes a machine-readable catalogue of 1146 spec-cited test cases against the server’s own wire, records every exchange, and computes verdicts as pure functions over what it recorded. Functional conformance, measured performance, step-load stress and the comparative-speed benchmark come from the one tool, and published results live on the public conformance and benchmark boards, submitted by pull request.

console.veredictum.eu is the official conformance instrument. It serves the catalogue and the vendored specification text, and a run driven there against a publicly reachable CDR is an official run: the record is submitted as a pull request, its verdicts are recomputed here from the transcript it carries, and it is signed only after they match. The instance holds no key and nothing durable — git is where a record lives. Running the instrument yourself publishes your own claim instead (Installation).

The released openEHR specifications are the only authority the instrument accepts. Every expectation in the catalogue names the section it comes from, so it can be refuted by a better reading of the specification and by nothing else.

Who this is for

• Evaluators and procurement teams. You need a claim about a CDR that someone other than its vendor can check. The instrument produces a conformance report, a statement of claims and a certificate, all three derived from a recorded run rather than asserted.
• CDR vendors and maintainers. You want to know where your server diverges from the released text before a customer finds out. A failing row arrives with the exchange that produced it and the specification section it violates.
• The openEHR community. The catalogue is an openEHR conformance test suite in machine-readable form. Where the specification is silent or contradicts itself, the divergence is recorded in an ambiguity register and reported upstream, never resolved privately.

What is in the box

Three commands, three stages

The pipeline splits into stages on purpose. Nothing is computed at a stage that could hide what an earlier stage recorded.

1. veredictum validate checks the catalogue itself. Zero findings is the only passing result, and it runs before any server is involved.
2. veredictum run drives the catalogue against your endpoints and writes results.json, a record of exchanges and not yet a judgement.
3. veredictum verdicts reads that record together with your statement of claims and writes the report, the statement and the certificate.

Installation covers getting the command. Running the instrument walks the three stages against a live server.

Coverage is machine-checked

A green run over a thin catalogue proves nothing, so a gate enumerates the wire surface from the released sources alone and fails on any behaviour that has neither a covering case nor a cited exception. Cases are added. They are never removed to make a run go green.

The conformance method says what the gate reads and what happens to a gap it finds.

Note

openEHR® is the registered trademark of the openEHR Foundation. Veredictum is an independent, community-driven conformance instrument: it names openEHR descriptively, to say what is being tested, and it is not an official openEHR Foundation product, not the Foundation’s CNF program, and not endorsed by or affiliated with the Foundation.

Installation

The zeroth way installs nothing: the official instrument at console.veredictum.eu browses the catalogue and drives runs against a publicly reachable CDR, and a run there produces an official record. Everything below is for running the instrument yourself, which publishes your own claim and works offline.

Four ways end at the same place: a veredictum you can point at a catalogue. Pick by what you already have installed. Whichever you pick, you also want a clone of the repository, and the next section says why.

• What you need, and why a clone is part of it
• With cargo
• With Docker: the web console
• From a release binary
• From source
• Checking that the install works

What you need, and why a clone is part of it

The instrument reads three things at run time: the catalogue, the vendored specification oracle, and your own declaration of the deployment under test. It reads all three as paths you pass on the command line, so the code and the data travel separately.

That split is deliberate. The catalogue and the specification oracle together are over 300 MB of data, which no package registry accepts, and an operator may legitimately want to point the instrument at a catalogue of their own. So the published crate and the published image carry the code, and a clone of the repository is where the data lives:

git clone https://github.com/Vernum-Projecten/Veredictum
cd Veredictum

The four ways below all run the same subcommands against that clone.

With cargo

The binary is on crates.io. Take this path if you want the command on your PATH.

cargo install veredictum --locked
veredictum validate --root artifacts --specs specs/openehr

--locked is worth understanding rather than copying: it builds against the Cargo.lock the release was tested with. Leave it off and cargo resolves fresh versions of every dependency, which is a different build from the one the project’s gates ran.

The library target is published with the binary, so you can consume the typed artifact model and the published JSON Schemas directly rather than reimplementing the format.

With Docker: the web console

The container image is the web console: a browser frontend over the same instrument, served by its own binary. The CLI is deliberately not distributed as an image: a static binary needs no container, and the release binaries below are its no-toolchain path.

The fastest start is the operator compose file, which pins the image to the version it shipped with and binds the console to loopback on port 3210:

curl -LO https://raw.githubusercontent.com/Vernum-Projecten/Veredictum/main/docker/docker-compose.yml
docker compose up

Open http://127.0.0.1:3210. Releases cut after v0.1.1 attach the same file to the release page, pinned to that release’s image. The equivalent docker run, if you would rather not use compose:

docker run --rm -p 127.0.0.1:3210:3000 -v "$PWD:/work" \
    ghcr.io/vernum-projecten/veredictum:<tag>

Substitute a published tag from the package page for <tag>. The image is multi-architecture and is pushed by digest, with its tags applied only after a smoke run and a vulnerability scan of that digest have passed.

The catalogue and the specification oracle are not baked into the image either, so the data you grade against is the data you can see in your own checkout: run compose beside a clone and the /work mount carries artifacts/ and specs/openehr/; run it beside an empty directory and the console comes up and says what it is missing. The console has no login, so both invocations above bind it to loopback; exposing it further is the operator’s decision, behind their own gate. The console chapter shows what it does today.

From a release binary

Prebuilt binaries for x86_64 and aarch64 Linux are attached to every release. Each tarball ships with a sha256sum, a CycloneDX dependency SBOM and a Sigstore bundle.

Verify the bundle before you run the binary. The check that matters is not just “this file is signed” but “this file was built by the workflow in this repository”, which is what --signer-workflow asserts:

gh attestation verify veredictum-<tag>-<target>.tar.gz \
    -R Vernum-Projecten/Veredictum \
    --signer-workflow Vernum-Projecten/Veredictum/.github/workflows/release-build.yml

A release is created as a draft and published only once every expected asset is attached, so you never meet a release whose binaries are still uploading. Its tag is signed, and a repository rule refuses to delete one, so a tag is never re-pointed at different code. The recovery path for a bad cut is the next version.

From source

The toolchain pins itself from rust-toolchain.toml, so you do not choose a Rust version:

cargo run -- validate --root artifacts --specs specs/openehr

The declared minimum supported Rust version is 1.96, verified in CI against that floor rather than assumed. The only extra tool is cargo-nextest, and only if you intend to run the project’s own test suite.

Checking that the install works

validate is the check to run first, because it needs no server:

veredictum validate --root artifacts --specs specs/openehr

A working install over an intact clone prints one line and exits zero:

1146 case(s), 249 binding(s), 43 capability row(s), 0 finding(s)

Any finding count above zero is a failure of the catalogue, not of your setup, and the findings printed above that line say which artifact is at fault. A missing --specs tree is the common first-run mistake: the citation and Service-Model gates are skipped without it, so the case count is reported while the checks that make the count meaningful never run.

What the instrument touches

You should not have to read the source to decide whether this tool is safe to run. This page says what each command reaches, and how to verify the binary before running anything.

Verify the binary first

Every release binary carries a Sigstore attestation. One command proves the exact bytes came out of this repository’s build workflow:

gh attestation verify veredictum-<tag>-<target>.tar.gz \
    -R Vernum-Projecten/Veredictum \
    --signer-workflow Vernum-Projecten/Veredictum/.github/workflows/release-build.yml

If that check fails, do not run the file.

What each command reaches

• bench sends HTTP requests to the base URL you give it, and writes into the --out directory you give it. Nothing else. The packs are compiled into the binary, so it fetches nothing; the credential is read from an environment variable and never rides argv; there is no telemetry and no other network destination. With --with-baselines it additionally drives your local docker CLI to pull and run the two pinned reference images (from ghcr.io and Docker Hub) on loopback ports, and removes them when done.
• run sends requests to the endpoints your own IXIT file declares, reads the catalogue and spec paths you pass, and writes into --out. It creates and deletes data on the target, which is why you point it at a test system.
• validate, verdicts, verify-record, bench-compare, the asset renderers are offline: they read the paths you pass and write where you say. No network at all.
• The console image binds loopback by default, has no login, and reaches a CDR only through the runs you start in it.

bench warns once when a credential would ride plain http to a host that is not loopback.

The record does not depend on trusting the operator

A published conformance reproduction is attested by the workflow that produced it, and a bench record embeds the histograms its summary is re-derived from, so both are checkable after the fact. The registry rules (registry/RULES.md) carry the details.

Running the instrument

• 1. Check the catalogue
• 2. Declare your deployment
• 3. Review your claim
• 4. Drive the catalogue
• 5. Compute the verdicts
• Measured performance, stress and speed
• Where to look when something goes wrong

A conformance campaign is four steps. Each writes a file the next one reads, so you can stop after any of them, inspect what was produced, and resume.

1. Check the catalogue

Before a server is involved, check the catalogue you are about to grade with:

veredictum validate --root artifacts --specs specs/openehr

Zero findings is the only passing result, and the command exits 1 when there is even one. The command reference lists the gates it runs.

Pass --specs every time. Without it the citation and Service-Model resolution gates do not run, and the case count still prints, which looks like a pass over a catalogue that was never fully checked.

2. Declare your deployment

The instrument needs to know where your server is and how to authenticate to it. That declaration is the IXIT file, and it is yours to write, against schemas/ixit.schema.json. Put it in a directory of your own beside the clone — mine/ixit.json in the commands below — with your claim next to it. fixtures/declaration/ carries both shapes filled in for a product that does not exist, which is a template to read and never a claim to copy.

This repository commits no declaration about anybody’s product. ISO/IEC 9646-7 splits an ICS proforma in one place: every cell belongs to the proforma specifier except the support and supported-values columns, which belong to the supplier of the implementation. The proforma is artifacts/vocab/capability_matrix.yaml, published here with a specification citation per row. The answers are yours.

The two files, and the split between them matters:

• ixit.json describes the deployment. Endpoints per instance, the authentication mode, and for a measured run an environment block naming the hardware and topology the numbers were produced on. Credentials are named, never carried: the file holds the names of the environment variables the instrument reads the user and password from, so no secret ever enters an artifact you might publish.
• statement.json is your declaration of claims. It names the product and version, the specification versions it targets, and the capabilities it claims to implement. The verdict machinery reads it as the thing being tested against the record.

A typical ixit.json declares three instances, because a full run needs to speak to the server as three different callers: an ordinary clinical user, an administrator, and no one at all. The unauthenticated instance is what lets the security cases check that a route refuses an anonymous request.

3. Review your claim

veredictum validate --root artifacts --specs specs/openehr \
    --statement mine/statement.json

--statement adds the static conformance review of ISO/IEC 9646-1 and -7 to the same command that checks the catalogue. It reports a capability you claim that the catalogue holds no verdict-bearing case for, a Signing claim your ixit declares no posture for, and a served-extension family the catalogue’s wire surface does not carry. Each of those would otherwise become a row that can never carry evidence, discovered mid-run.

4. Drive the catalogue

veredictum run --root artifacts --ixit mine/ixit.json --out out/ \
    --sut-name my-cdr --sut-version 1.2.3 --statement mine/statement.json

The command drives every applicable case against your endpoints and writes out/results.json, a record of what was sent, what came back, and how it was classified. It also writes out/run-exceptions.json, which lists the cases the interpreter could not drive at all.

results.json is a record, not a judgement. That separation is the point: the recorded exchange stays available for anyone to re-read, and the judgement is computed from it in a separate step that touches no network.

Passing --statement here changes what runs. A case gated on an option your statement does not declare is recorded as not-applicable at drive time rather than driven, which is the test-selection discipline ISO/IEC 9646 describes. Omit the flag and everything is driven, which is what you want when you are exploring an unfamiliar server rather than grading a declared one.

--filter takes a substring matched against case identifiers, which is how you re-drive one chapter while working on a fix. The resulting results.json holds only the cases that ran, so a filtered run is a working tool and never the record you submit.

The command exits 1 if any case failed or errored, so a shell script can gate on it.

Note

A functional run never re-measures. If a results.json already exists at the --out path, its measurement records are carried forward, so running the functional catalogue again after a measured run does not discard the performance evidence. A file that is present but unreadable stops the run instead, because carrying zero measurements past it would drop that evidence silently.

5. Compute the verdicts

veredictum verdicts --root artifacts --statement mine/statement.json \
    --results out/results.json --out out/

This step is a pure function of the statement, the recorded results, the catalogue and the catalogue’s capability matrix. Run it twice on the same files, on any machine, and you get the same bytes out. It writes:

Nothing in these documents is asserted by hand. A number that appears in them was computed from the record in the same run that printed it.

Measured performance, stress and speed

Those four steps cover functional conformance. Three more instruments produce the other kinds of evidence.

• veredictum perf earns a volumetric class (POC, S, L or R) with an open-loop measured run and merges the measurement record into an existing results.json. The normative window is one hour; --hours extends it to 2, 4, 6, 8 or 12, which is a stricter demonstration. Nothing shorter than the case exists.
• veredictum stress climbs geometric load steps to find where the deployment breaks and writes stress.json. It is exploration only and is never a conformance record, which is why it is class-free by design.
• veredictum bench measures comparative speed against any reachable CDR and writes a bench-result document. It needs no artifact root, no IXIT and no statement, and it is a speed record and never a conformance record. The benchmark board ranks the submitted ones.

perf and stress both need the environment block in your IXIT file filled in, because a throughput number without the deployment described says nothing. All three want an idle machine and a deployment whose resource limits match the envelope you are claiming. A measured run on a laptop that is also running a browser measures the browser.

Where to look when something goes wrong

A red row is not presumptive evidence of a bug in your server. It is evidence that the specification, the catalogue and the server do not all three agree, and which one is wrong is a question that gets answered before anything is changed. The conformance method sets out how, and what evidence an attribution has to carry.

The command reference lists every subcommand with its real flags.

Command reference

• emit-schemas
• run
• validate
• perf
• stress
• bench
  • The embedded packs
  • Posture profiles
  • Posture canaries
  • Same-machine baselines
  • The relative index
  • Submittability
• bench-compare
• bench-packs
• aql-probe
• stress-compare
• perf-assets
• conformance-assets
• verdicts
• replay
• evidence
• verify-record

Every code block below is veredictum <command> --help from the binary itself: this page is rendered by scripts/render/commands-md.sh out of clap’s own output, so a flag cannot land in the binary and go missing here. The sections follow the order veredictum --help lists them in, and the prose under each one is written by hand and lives in website/book/commands/.

Three commands make the conformance record (validate, run, verdicts), three measure (perf, stress, bench), replay re-judges a recorded run out of its own transcript, evidence carves the exchanges behind a red run’s rows out of that transcript, verify-record checks a sealed bundle, and the rest render or explore.

emit-schemas

Write the published JSON-Schema set (byte-deterministic)

Usage: veredictum emit-schemas --out <OUT>

Options:
      --out <OUT>  Output directory (created if missing)
  -h, --help       Print help

Byte-deterministic. The schemas in the repository’s schemas/ directory are this command’s output, drift-tested against it, which is how the published format and the code that reads it stay one thing. Author against these if you are writing your own catalogue or your own harness.

run

Execute the catalogue against a live SUT (from its ixit topology) and emit results.json + the run report

Usage: veredictum run [OPTIONS] --root <ROOT> --ixit <IXIT> --out <OUT>

Options:
      --root <ROOT>
          The artifact root

      --ixit <IXIT>
          The ixit topology file (JSON)

      --out <OUT>
          Output directory for results.json + the run summary

      --sut-name <SUT_NAME>
          SUT display name
          
          [default: ferroehr]

      --sut-version <SUT_VERSION>
          SUT version label
          
          [default: dev]

      --filter <FILTER>
          Only run cases whose id contains this substring

      --statement <STATEMENT>
          The party statement (ICS) — the list ISO/IEC 9646 test selection selects from. Supplied, an option-gated case whose arm the ICS does not declare is recorded N/A at drive time; absent, NO arm of a mutually exclusive branch is selected, so every such case and every extension route is recorded N/A with its citation and the run says so once

      --sign-key <SIGN_KEY>
          An armored OpenPGP secret key — when supplied, the emitted documents are sealed with `record-manifest.json` and its detached signature, which `verify-record` and `gpg --verify` both check

      --sign-passphrase <SIGN_PASSPHRASE>
          The passphrase unlocking `--sign-key`. Supply it through the environment: a passphrase on the command line is visible to every process on the host
          
          [env: VEREDICTUM_SIGN_PASSPHRASE]

      --record-exchanges
          Persist the wire exchanges beside results.json as transcript.json.
          
          OFF by default. The artifact can carry real patient data — a SUT's response body is recorded verbatim — so it is operator-controlled output, never a log: store it where the record itself is stored. The `authorization` request header's value is withheld; everything else lands exactly as it went out and came back.

      --progress
          Print one machine-parseable line per processed case: `progress: <k>/<n> <case-id>` after `progress: 0/<n>` — a stable grammar a driver may parse. Off by default, so existing output is byte-identical without it

  -h, --help
          Print help (see a summary with '-h')

Drives every applicable case and records the exchange. Exits 1 if any case failed or errored.

validate

Validate one artifact tree through every machine gate

Usage: veredictum validate [OPTIONS] --root <ROOT>

Options:
      --root <ROOT>
          The artifact root (schedule/, bindings/, vocab/, corpus/, registers/)

      --specs <SPECS>
          The vendored openEHR spec tree; enables SM-operation and spec-ref resolution

      --write-report
          Also refresh `<ROOT>/coverage-report.md` from `--specs`.
          
          OFF by default: `validate` is a check verb, and a check that mutates the working tree is a trap for read-only and fenced invocations. The pipeline scripts that publish the report pass this explicitly.

      --statement <STATEMENT>
          A submitted declaration (`statement.json`) to hold to the static conformance review, with the `ixit.json` beside it.
          
          ISO/IEC 9646-7 assigns an ICS proforma's support and supported-values columns to the supplier of the implementation, so no declaration is committed here and none is swept from the tree.

  -h, --help
          Print help (see a summary with '-h')

Every machine check over the catalogue: identifier uniqueness, citation resolution, binding completeness, coverage of the enumerated wire surface, and the per-capability case-count floors. It prints one line per finding and a summary line, and exits 1 if the finding count is not zero.

Supplying --statement adds the static conformance review of ISO/IEC 9646-1 and -7 over that one declaration: a claimed capability the catalogue holds no verdict-bearing case for, a Signing claim the ixit beside it declares no posture for, and a served-extension family the catalogue’s wire surface does not carry.

--write-report is off by default on purpose. A check verb that rewrites a file on every run is a trap for read-only invocations, so the pipelines that publish the coverage report ask for it explicitly. The report lands at <ROOT>/coverage-report.md, beside the artifact families it measures.

perf

Execute the performance schedule's open-loop measured run(s) against a live SUT and merge the §8.10 measurement records into an existing results.json (conformance-by-measurement)

Usage: veredictum perf [OPTIONS] --root <ROOT> --ixit <IXIT> --results <RESULTS> --class <CLASS>

Options:
      --root <ROOT>                  The artifact root
      --ixit <IXIT>                  The ixit topology file (JSON) — its environment block is mandatory for a measured run
      --results <RESULTS>            The results.json to merge the measurement records into (written by a prior `run`)
      --class <CLASS>                Select the performance case(s) of this class (POC | S | L | R)
      --seed-workers <SEED_WORKERS>  Parallel seeding workers [default: 16]
      --hours <HOURS>                The sustained-window ladder: hours to hold the offered load — 1 (default, the case's normative window) | 2 | 4 | 6 | 8 | 12. A longer window is a STRICTER demonstration and persists like any measured run; nothing shorter than the case exists [default: 1]
  -h, --help                         Print help

Conformance by measurement. Latency is measured from the planned arrival instant under open-loop offered load, so a stall shows up as latency instead of disappearing into a slowed-down request rate. A longer --hours window is a stricter demonstration and persists like any measured run; nothing shorter than the case exists.

The environment block is mandatory rather than optional because a latency number is a claim about a deployment, and a claim with no deployment described cannot be checked or reproduced.

stress

Run the step-load STRESS instrument: geometric load steps to the maximum sustainable throughput — where the system breaks (exploration only; never a conformance record, and class-free by design)

Usage: veredictum stress [OPTIONS] --root <ROOT> --ixit <IXIT> --out <OUT>

Options:
      --root <ROOT>                  The artifact root
      --ixit <IXIT>                  The ixit topology file (JSON) — its environment block is mandatory (a throughput number without the deployment described is meaningless)
      --out <OUT>                    Where to write the stress report (stress.json)
      --corpus-class <CORPUS_CLASS>  The class-scale corpus the stress runs on (POC | S | L | R — the standardized corpus selector): data volume + workload mix only; no class floor enters the stress report or chart [default: POC]
      --seed-workers <SEED_WORKERS>  Parallel seeding workers [default: 16]
      --step-secs <STEP_SECS>        Each load step's recorded hold, seconds (short + intense by design) [default: 120]
      --bisections <BISECTIONS>      Post-breach bisection refinements [default: 3]
      --max-rate <MAX_RATE>          The climb cap (arrivals/s) [default: 4096]
  -h, --help                         Print help

Exploration only, and class-free by design: no class floor enters the stress report or its chart. A stress.json is never a conformance record, and quoting one as if it were is a misuse of the tool.

bench

Run the universal SPEED benchmark: an embedded pack against any reachable CDR over its base URL, seeded once and measured N times open-loop (comparative speed only; never a conformance record, a certificate, or a performance-class rating)

Usage: veredictum bench [OPTIONS] --base-url <BASE_URL> --out <OUT>

Options:
      --base-url <BASE_URL>
          The system's base URL, up to and including the openEHR REST base (for example `https://cdr.example/openehr/v1`)

      --auth <AUTH>
          How the client presents itself: `none`, `basic` or `bearer`.
          
          The secret never rides argv: `basic` reads `VEREDICTUM_BENCH_PASSWORD` and `bearer` reads `VEREDICTUM_BENCH_TOKEN`.
          
          [default: none]

      --user <USER>
          The user `--auth basic` presents

      --pack <PACK>
          The embedded pack to drive
          
          [default: smoke]

      --posture <POSTURE>
          The posture profile to declare, out of the set the pack defines. Omit to take the pack's first, which is always `minimal`.
          
          The run's canaries check the declaration against the running system before and after the measured window, and a contradiction refuses the run.

      --repetitions <REPETITIONS>
          How many times to repeat the measured phases. A result with fewer than three is recorded as not submittable, and names that as one of its unmet requirements
          
          [default: 3]

      --scale <SCALE>
          Multiply the pack's EHR count by this factor, for a shorter run. Anything but `1.0` takes the run off the pack's pinned configuration, and the record says so
          
          [default: 1]

      --seed-workers <SEED_WORKERS>
          Override the worker count every seed phase declares. Omit to run the pack's own value, which is what its reference figures describe

      --with-baselines
          After the target's run, compose each pinned reference CDR on this host from its own digest-pinned images, drive the same pack at the same seed against it, and record the relative index. Needs the docker CLI; a record without a baseline is not submittable

      --out <OUT>
          Output directory for the result document and its summary

      --label <LABEL>
          A label for this run, which names its column in a comparison and distinguishes its file name in a shared output directory

  -h, --help
          Print help (see a summary with '-h')

A bench run needs no artifact root, no IXIT and no party statement. The pack is compiled into the binary with a sha256 pin on every fixture, and the pins are recorded in the result.

The embedded packs

smoke proves the engine: one blood-pressure template, a small EHR corpus, and one mixed open-loop phase over the read, write and query surface.

community-vitals reproduces the openEHR community’s vital-signs benchmark harness (thread 17224) and measures the same work a second way. Its write phase creates 100 EHRs and commits the same Vital signs composition 1,000 times into each with Prefer: return=identifier, on one worker, reporting bulk-load throughput plus the whole-loop milliseconds-per-composition average the thread quotes. Its read phase then runs twice over that population. read_walk is the sequential walk the harness performs, seven GETs against every committed composition: the latest version, the same at an instant, the VERSIONED_COMPOSITION, its latest version, its version at that instant, one version by id, and the revision history. It reports the whole-loop microseconds-per-request average. read_open_loop offers the same seven reads as an arrival schedule pinned at 200/s for 60s after a 15s warmup, which is where the coordinated-omission-free percentiles come from. The pinned rate is part of the pack version.

Every number in the record carries the discipline that produced it: a closed-loop whole-loop average and an open-loop percentile answer different questions and are never read against one another. The two fixtures are embedded byte-identically, the operational template from the vendored CKM export for template id Vital signs and the composition from the attachment on post 8 of that thread, both pinned by sha256 and verified at load.

aql-mix measures query speed over that same population, from the same two pinned fixtures, so a query figure and a read figure describe the same corpus. Its seed phase creates 50 EHRs and commits the composition 20 times into each, on a pool of 8 workers. The pack version pins that population, and it is sized for query shapes: large enough that a query has to choose an access path, small enough to load before a measured window opens. The measured phase is open-loop at 24 arrivals a second for 60s after a 15s warmup, over six query classes at equal share, so each class is offered at 4 arrivals a second and every class returns the same number of samples.

Each class posts its own AQL statement to /query/aql, accepts only 200, and counts every other answer in its own error class, so a server that refuses one shape leaves the other five classes’ percentiles alone. The systolic threshold, the page offset, and the EHR or composition each arrival addresses all draw from the run’s seeded streams, so no arrival repeats the previous one’s result set and the whole draw is reproducible from the seed the record discloses. Why each class exists travels with the pack definition, so a rendered view explains a column without knowing anything about the pack’s internals.

--scale shrinks the EHR count for a quick run and changes nothing else. Anything but 1.0, or a --seed-workers override, takes the run off the pack’s pinned configuration; the record says so in its scale block, the summary says so in prose, and bench-compare names it in the header.

Credentials never ride the command line: --auth basic reads its password from VEREDICTUM_BENCH_PASSWORD and --auth bearer reads its token from VEREDICTUM_BENCH_TOKEN.

Before anything is measured, a preflight reads the template list, uploads the pack’s template, then creates one scratch EHR, commits a composition into it and reads that composition back. A failure at any of those refuses the run and names the exchange, so a half-measured document never exists.

The run then seeds its corpus once and repeats the measured phases. Measured phases are open-loop: arrivals fire at their planned instants whatever the system is doing, and every latency is measured from the planned instant, so a stall shows up in every arrival queued behind it.

Posture profiles

A CDR running without an audit trail and unsigned versions is a different system from the same CDR with both switched on, and a speed number that does not say which one it measured cannot be compared with anything. Every pack therefore defines named posture profiles, and a run declares exactly one.

minimal is the bare spec-conformant surface, and every pack defines it: no audit trail, unsigned versions, commits validated against the operational template, uncompressed responses, one tenant. Validation sits at template rather than at nothing because the specification puts it there: ITS-REST specifications/responses/422.yaml defines the commit refusal as the case where the template “is not validating the supplied resource”. A server that accepts anything is below the floor rather than lightly configured. community-vitals also defines clinical-default, which is the same surface with an audit trail written.

The record’s posture block carries the profile, its summary, and one line per disclosed item: the audit sink, the version-signing scheme, the commit-validation depth, the authentication mode, TLS, response compression and tenancy. Each item is a closed vocabulary, so an unknown token is refused rather than read as a default.

Posture canaries

A declaration is a promise, so each item is also probed black-box and labelled verified or declared-only, with the exchange behind the label recorded beside it.

The canaries run before the measured window and again after it. A reading that contradicts the declaration refuses the run, naming the item, the declared value, the observed one and the exchange. So does a pair of brackets that disagree with each other, because a posture that moved mid-run leaves the numbers straddling two systems. Neither is recorded as a footnote beside a published figure.

Baselines run under the profile the target declared and carry their own verified block, so both sides of a ratio disclose what was switched on behind them. bench-compare states a posture disagreement between columns in the header, above the numbers.

Same-machine baselines

--with-baselines anchors the run. After the target’s own measurement, the instrument composes each pinned reference CDR on this host, drives the same pack at the same seed for the same number of repetitions against it, and tears the stack down with its volumes so the next baseline starts from an empty database. The record then carries a baseline block per reference, each a full per-operation summary exactly like the target’s, beside the digest-pinned images, the upstream deployment recipe the topology follows, and the container ceilings both stacks ran under. Every image is pinned by digest rather than by tag, so two submitters measure the same bytes.

The flag needs the docker CLI. On a host where it does not answer the run is refused before the target is touched, with the missing binary named; a run without the flag needs no container runtime at all.

The relative index

An absolute millisecond describes the system and the machine together, so two records taken on different hosts cannot be read against one another. The relative index can. For every phase, operation and metric it is the target’s cross-repetition median divided by the baseline’s, both measured on the same host in the same session, so the machine cancels. On a latency metric a value below 1.0 means the target answered faster than the baseline and above 1.0 slower; on throughput the sense inverts, because there a larger number is the faster system. Each ratio is serialized with the two medians it came from.

Where no ratio can be formed the record says so and why: an operation only one side measured, a phase only one side ran, or a baseline median of zero. A gap is recorded rather than omitted, because a missing row in a comparison reads as agreement.

Submittability

A record is submittable when it carries at least three repetitions, at least one same-machine baseline, and no operation that lost more of its arrivals than the pack’s ceiling allows. That ceiling is part of the versioned pack definition and the record discloses it as pack.max_failed_share; every embedded pack pins 0.01. The check reads every repetition, phase and operation, on the target and on every baseline block, because an index divides by a baseline median and a divisor taken from arrivals that failed describes the failure. A record that misses any requirement stays valid for local exploration and names the ones it misses, in the submittable_unmet list, in the rendered summary and in every bench-compare column header.

The rendered summary carries a failed-arrival table: one row per side, repetition and phase, with the phase’s own share and the worst operation inside it, followed by a sentence per reading above the ceiling naming where it went. The environment fingerprint prints on the summary header and in every comparison column, so no number is read without the machine it came from.

A bench result is a benchmark record for comparative speed. It is not a conformance record, not a certificate, and not a performance-class rating; a bench result may motivate a class run, never substitute for one.

bench-compare

Align two or more committed bench results into one table, one column per file, with every pack or host mismatch stated in the header

Usage: veredictum bench-compare --result <RESULTS> --out <OUT>

Options:
      --result <RESULTS>  A committed bench result. Repeat the flag once per file; at least two are needed
      --out <OUT>         Output directory for the rendered comparison
  -h, --help              Print help

One column per file, one row per phase, operation and metric. Each cell carries the cross-repetition median with the inter-quartile range beside it, so a reader sees the spread as well as the number.

Every column header carries the machine the run was generated on, the worst failed-arrival share the run recorded beside the ceiling its pack pins, and a column that is not submittable names the requirements it misses rather than printing one bare false. Where the columns carry a relative index, it gets its own table below the header, at p50 and p99 per baseline: that is the part which survives a change of host.

A mismatch is stated above the table, never under it: columns that ran different pack versions, columns generated from different hosts, columns that ran at different scale factors or off the pack’s pinned configuration, and columns whose runs are not submittable are all named in the header, and the command exits 1 when any of them applies. Columns from different hosts where at least one carries no relative index are named too, because nothing in that table is comparable across them. Each row also names the discipline its numbers came from.

bench-packs

Write the embedded benchmark pack manifest: every pack's phases with their load discipline, its operation mix with what each entry probes, its posture profiles, its fixture pins with their provenance, its seed, and the submission requirements (byte-deterministic)

Usage: veredictum bench-packs --out <OUT>

Options:
      --out <OUT>  Output directory (created if missing)
  -h, --help       Print help

A pack is versioned data compiled into the binary, so the binary is the only honest source for a description of one. This command writes that description: per pack the id, the version, the seed every arrival stream draws from, the failed-arrival ceiling a record is judged against, each phase with its load discipline and its counts, each measured phase’s operation mix with the share and the probe rationale of every entry, each posture profile the pack defines with what it declares item by item, and each embedded fixture with its sha256 pin, its size and where the bytes came from. The document also carries the boundary statement, the methodology, how a relative index is derived, what the seed and the posture canaries govern, and the requirements a record meets before it may be ranked.

Emission is byte-deterministic and every collection is ordered, so regenerating the file and diffing it is a build gate. The public page at veredictum.eu/benchmark-methodology.html is generated from the committed copy of this document, and CI refuses a pack change that leaves either of them stale.

aql-probe

Run the AQL optimization probe: fire the measurement machinery's AQL vocabulary against a live, freshly seeded SUT, record wire percentiles, and attribute the DB-side cost per probe via `pg_stat_statements` (exploration evidence for the optimization loop — never a conformance record)

Usage: veredictum aql-probe [OPTIONS] --root <ROOT> --ixit <IXIT> --out <OUT>

Options:
      --root <ROOT>                  The artifact root
      --ixit <IXIT>                  The ixit topology file (JSON) — the `containers` block enables DB-side attribution and maintenance settling
      --out <OUT>                    Where to write the probe report (aql-probe.json)
      --corpus-class <CORPUS_CLASS>  The class-scale corpus the probes run against (POC | S | L | R) [default: POC]
      --seed-workers <SEED_WORKERS>  Parallel seeding workers [default: 16]
      --requests <REQUESTS>          Requests fired per probe [default: 20]
  -h, --help                         Print help

Fires the measurement machinery’s AQL vocabulary, records wire percentiles per probe, and attributes the database-side cost through pg_stat_statements. This is evidence for someone optimizing a server, and it is never a conformance record.

stress-compare

Render the cross-SUT stress overlay (both systems' latency-throughput curves on one canvas) FROM two committed stress reports — deterministic, both directions on equal footing

Usage: veredictum stress-compare --left <LEFT> --left-label <LEFT_LABEL> --right <RIGHT> --right-label <RIGHT_LABEL> --out <OUT>

Options:
      --left <LEFT>                The primary SUT's committed stress.json
      --left-label <LEFT_LABEL>    The primary SUT's display label
      --right <RIGHT>              The comparison SUT's committed stress.json
      --right-label <RIGHT_LABEL>  The comparison SUT's display label
      --out <OUT>                  Where to write the overlay SVG
  -h, --help                       Print help

Deterministic, and both directions on equal footing: the two curves are drawn by the same code from the same kind of file, so neither side gets a rendering advantage.

perf-assets

Render the published performance SVG assets FROM a committed results.json (deterministic; CI regenerates and diffs — hand-drawn numbers are a build failure)

Usage: veredictum perf-assets [OPTIONS] --root <ROOT> --results <RESULTS> --out <OUT>

Options:
      --root <ROOT>        The artifact root (for the class-ladder floors)
      --results <RESULTS>  The committed results.json carrying the measurement records
      --out <OUT>          Output directory for the SVG files
      --summary <SUMMARY>  Also write the generated Markdown summary (class ladder + measured detail) to this path — the book's build-time include
      --stress <STRESS>    A committed stress report (stress.json) to render the latency-throughput curve from, when one exists
  -h, --help               Print help

conformance-assets

Render the conformance visuals (the capability heat grid + the per-chapter outcome bars) deterministically FROM the committed party artifacts — the perf-assets pattern for functional conformance

Usage: veredictum conformance-assets [OPTIONS] --root <ROOT> --results <RESULTS> --verdicts <VERDICTS> --out <OUT>

Options:
      --root <ROOT>          The artifact root (for the capability matrix)
      --results <RESULTS>    The committed results.json
      --verdicts <VERDICTS>  The committed verdicts.json
      --out <OUT>            Output directory for the SVG files
      --suffix <SUFFIX>      A suffix appended to the SVG file stems (`-ehrbase` for the comparison SUT's copies) [default: ""]
  -h, --help                 Print help

Both renderers are deterministic over files already committed, so a build job can regenerate them and diff the result. A hand-drawn number in a published chart is a build failure rather than a review comment.

verdicts

Compute the verdicts from a statement + results against an artifact tree (the pure pipeline) and write the rendered submission documents

Usage: veredictum verdicts [OPTIONS] --statement <STATEMENT> --results <RESULTS> --root <ROOT> --out <OUT>

Options:
      --statement <STATEMENT>
          The party statement (`statement.json`)
      --results <RESULTS>
          The party results (`results.json`)
      --root <ROOT>
          The artifact root (schedule/, vocab/, registers/)
      --out <OUT>
          Output directory for the rendered documents + verdicts.json
      --sign-key <SIGN_KEY>
          An armored OpenPGP secret key — when supplied, the rendered documents are sealed with `record-manifest.json` and its detached signature, which `verify-record` and `gpg --verify` both check
      --sign-passphrase <SIGN_PASSPHRASE>
          The passphrase unlocking `--sign-key`. Supply it through the environment: a passphrase on the command line is visible to every process on the host [env: VEREDICTUM_SIGN_PASSPHRASE]
  -h, --help
          Print help

The pure step. It reaches no network and reads nothing but its inputs, which is what makes a published verdict re-derivable by anyone who has the same four files.

replay

Re-judge a recorded run from its transcript, answering every composed request out of the recording instead of a server, and optionally refuse a results document the recorded exchanges do not support

Usage: veredictum replay [OPTIONS] --root <ROOT> --ixit <IXIT> --transcript <TRANSCRIPT>

Options:
      --root <ROOT>              The artifact root (schedule/, bindings/, corpus/, vocab/)
      --ixit <IXIT>              The ixit topology the recorded run was driven under
      --transcript <TRANSCRIPT>  The transcript of that run (`transcript.json`)
      --statement <STATEMENT>    The party statement ISO/IEC 9646 test selection re-applies: it decides which option arm, extension route, claimed capability and release floor the re-judgement selects. Omitting it re-derives a whole-catalogue sweep stamped `selection_basis: statement_blind`, which `--against` refuses to hold against a record an ICS selected
      --filter <FILTER>          Re-judge only cases whose id contains this substring
      --out <OUT>                Where the re-judged `results.json` is written
      --against <AGAINST>        The submitted `results.json` the re-judgement is held against: any row whose status or row counts differ fails the command
      --progress                 Print `progress: <k>/<n> <case>` lines while re-judging
  -h, --help                     Print help

Only the transport changes. The catalogue is driven again through the same request composition, the same response classification and the same assertion evaluators the live run used, with the recorded response standing in for the server’s. A case whose recording runs out, or whose replay composes a request the recording does not carry, records a transport failure: a verdict is never reproduced over evidence nobody has.

With --against, every row is compared on its status and its two row counts, and any disagreement exits 1 naming the case. The reason text is not compared, because a replay reaches a recording rather than a server and identical judgements can carry different words.

Omitting --statement re-derives a sweep of the whole catalogue. The replay says so on stderr, in the words a live run uses, and stamps selection_basis: statement_blind on the document it writes. With --against the selection facts a results.json records are compared before any row is: a record an ICS selected, re-judged blind, under a statement the record does not name, or under one declaring different its-rest formats, exits 2 rather than reporting agreement, because a re-derivation under another claim re-derives another campaign. The statement is named by statement_digest, the leading 8 bytes of the SHA-256 over the declaration’s own bytes, so the refusal prints the recorded value and the applied one and a reader checks either with sha256sum statement.json | cut -c1-16. A record written before selection_basis or statement_digest existed identifies nothing about what selected it, and the replay reports that instead of refusing.

What this establishes is that the judgement follows from the evidence. It does not establish the evidence: a transcript is what the instrument says it sent and received.

evidence

Export a finished run's recorded exchanges for a named set of cases: the triage input, carved out of the run's own transcript. No statement is read, and a bundle that would carry nothing is refused

Usage: veredictum evidence [OPTIONS] --transcript <TRANSCRIPT> --out <OUT> <--failing|--only <ONLY>|--filter <FILTER>>

Options:
      --transcript <TRANSCRIPT>  The run's `transcript.json` (written by `run --record-exchanges`)
      --results <RESULTS>        The run's `results.json`. Required by `--failing`, and otherwise optional: supplying it puts each exported case's outcome row beside its exchanges
      --failing                  Export the red rows the results record names — every `failed` and every `errored` case. The one-command triage input
      --only <ONLY>              Export this case, by id. Repeat the flag once per case
      --filter <FILTER>          Export cases whose id contains this substring
      --out <OUT>                Where the bundle is written
  -h, --help                     Print help

At least one of --failing, --only and --filter is required, and the three union: a case is exported when any of them names it. The unfiltered document is the transcript itself.

The red rows of a run become a triage input in one command:

veredictum evidence --transcript run/transcript.json \
    --results run/results.json --failing --out run/evidence.json

No statement is read. Sealing a record needs a claim; reading the exchanges a run recorded does not, and a run that went red is exactly when they are needed.

An export that would carry nothing is refused, exit 2, with no file written. A selection matching no recorded case names what was asked for and what the transcript actually carries; a selection whose every case recorded no exchange names those cases and says that recording is opt-in. A selection that half-matched still exports, and the bundle’s without_exchanges names every case it could not carry, so a partial answer never reads as a complete one.

The authorization request header’s value is withheld by the export itself, whatever the transcript held. Response bodies are the wire’s own bytes and can carry real patient data, so the bundle is operator-controlled output like the transcript it comes from.

verify-record

Verify a sealed bundle: recompute every digest its record manifest names, and check the detached signature over that manifest

Usage: veredictum verify-record --record <RECORD> --key <KEY>

Options:
      --record <RECORD>  The bundle directory holding the emitted documents, the record manifest and its detached signature
      --key <KEY>        The armored OpenPGP public key the signature is checked against
  -h, --help             Print help

Prints the signer fingerprint, the signing time, and one line per file with its digest verdict. Zero findings is the only passing result: a digest mismatch, a file the manifest names but the bundle does not carry, or a signature no component of the supplied key verifies, each exits 1 naming what failed.

The bundle is ordinary files, so the check does not depend on this tool. gpg --verify record-manifest.json.asc record-manifest.json establishes the same signature, and sha256sum re-derives the same digests.

A verified bundle is one link in the chain and not the whole of it. A valid signature proves integrity and origin since signing, and says nothing about the conditions the run executed under. The published instrument, the verification pack and the citation-carrying record are the rest, which is why that sentence prints on every verification.

The web console

The console is the instrument with a browser in front of it. It ships as its own container image, reads the catalogue and the vendored specification text from paths you mount, and reaches the engine only through the published veredictum crate: every number it shows is one the command line prints too.

Every screenshot below is captured by the console’s own browser journeys (scripts/ui-e2e.sh with UI_E2E_DOCS_SHOTS=1), in one 1440×900 browser window, light and dark. They are refreshed in the pull request that changes the interface, so what you see here is the interface that shipped.

A capture pass serves the console in capture mode, where the three facts one run stamps show fixed stand-ins: the run clock reads 00:00, a record digest reads as all zeros, and a signing time reads 1970-01-01T00:00:00Z. Your own runs show the real values, and a sealed record always carries them: the mode changes what the screen displays, never what is written, sealed or signed. It exists so a capture pass over an unchanged interface produces identical images, which is what makes a screenshot diff mean something.

The landing

The four counts are the catalogue’s own: case cores, operation bindings, capability-matrix rows, and validate findings. A findings count above zero means the catalogue itself needs attention before any server is graded.

The catalogue explorer

Chapters first, each with the number of cases it carries. The search and page state live in the URL, so a view is shareable and survives a refresh.

One chapter lists its cases: the identifier, the kind, and the test purpose. One behaviour per case, so a red row names one defect.

One case in full

The case detail carries the description, the specification citations the expectation stands on, the operation bindings that realize it on the wire, and the corpus fixtures it uses. The citation list is the point: an expectation is refuted by a better reading of the cited text, and by nothing else.

The run wizard

Grading starts at Connect: the CDR base URL, the display identity, the authentication choice from the ixit’s own vocabulary, and a probe whose answer renders verbatim before anything continues.

Scope takes the claim the run grades: the vendor’s own statement.json (the ICS, the document that says which profiles and capabilities are claimed), pasted into the box. The console offers no declaration of its own to load, because the claim belongs to whoever makes it: ISO/IEC 9646-7 assigns an ICS proforma’s support and supported-values columns to the supplier of the implementation, and the proforma itself is the capability matrix this instrument publishes.

The claim can also be built on the screen. The tier row offers CORE, STANDARD, OPTIONS and SEC-BASIC with the capabilities the capability matrix puts in each tier and the number of catalogue cases those capabilities gate, counted through the same matrix walk the judgement computes each profile verdict from. Composing writes that claim into the same box, product identity taken from the Connect step, so the operator reads the exact document the run will be graded against. Option branches stay undeclared there: only the party running the server knows which branch it realizes.

The document is held to the published statement schema before anything is stored, and saving answers with the claim overview (product, claimed tiers, capability count) beside the selection preview, so the screen says what will run before anything starts. An empty box is an honest no-claim run. The verdict later certifies exactly the pasted claim against the recorded evidence.

Scope also decides whether the run keeps its wire. “Record the wire exchanges” is off by default; ticked, the run writes transcript.json beside its results.json, carrying every request and response it drove. The artifact records a server’s response bodies verbatim, so it can hold real patient data from the deployment being graded: it lands in the operator’s own output directory, the sealed record covers it, and the authorization request header’s value is withheld.

The live screen renders the engine’s own progress stream: the case counter, the elapsed clock, a moving-median estimate labelled as such, and the tail of the engine’s output. When the run finishes the outcome links straight to the record.

The record: results and verdicts

Results reads the finished run’s own record, red rows first. A row links to its detail: the recorded reason beside the case’s specification citations, which is what the attribution of that row turns on.

The detail ends with the wire. A run driven with the recording box ticked shows each exchange as its request and response panes, verbatim, which is where a triage starts. A run driven without it says so in the same place.

Verdicts is the same pure function the command line runs, over the same record: the profile matrix with its coverage bounds printed, and the rendered documents byte-for-byte.

The export: one signed record

A verdict nobody can check is an opinion. The export step on the verdicts screen hands the run to the pinned instrument’s own verdicts --sign-key, which writes the rendered documents, a digest manifest over them, and a detached OpenPGP signature over that manifest. The console adds no sealing of its own; it supplies the key path and reads the result back.

Beside the sealed set the console renders three files a party publishes. Each carries the record digest prefix, so the artwork names the bytes it certifies rather than being a logo anyone could copy:

• The seal card is the brand’s certificate master with its three slots filled: the product under test, the profile verdict, and the moment the verdict was spoken, read back from the signature’s own creation time because the record carries no wall clock.
• The badge is a compact SVG for a README, with the digest prefix in its title and the verify path in its source.
• The report is one self-contained HTML file of everything the results and verdicts screens show, with the full digest, the signer fingerprint and the signing time in its footer.

All three are pure functions of the record: the same bundle in produces the same bytes out, which is what lets anyone re-render them and compare.

The seal card, the badge and the report sit deliberately outside the manifest. The manifest covers what the instrument judged; these are renderings about that judgement, and signing a rendering of a rendering would say nothing extra.

Two keys are configuration. VEREDICTUM_SIGN_KEY names the armored secret key the bundle is sealed with, and VEREDICTUM_VERIFY_KEY names the public half. The console asks for both because it will not print a signing time it has not checked: after sealing, it verifies its own bundle before stating who signed it and when. With neither mounted the section says so and offers no button.

Submit: the run publishes itself

A finished run at the hosted instrument becomes a published record by being committed to the public results registry. The submit step asks for the disclosure the submission rules make mandatory and states everything the run already knows.

The instrument drove an endpoint you named, so it can say what that server answered and nothing about the host behind it. That is the split the form is built on. The run supplies the endpoint it drove, the moment it started, the catalogue revision its results record names, and the engine version the console links. You supply who is publishing, the machine the graded server runs on, what was switched on behind the result, and the conflict-of-interest sentence the rules give no “not applicable” for. An empty mandatory field is refused by name before anything is opened.

The submission adds one entry and the five record files a re-derivation reads: the results, the verdicts, the recorded exchanges, the topology they were driven under and the claim they were judged against. Each is listed by role with the SHA-256 of the exact committed bytes. The entry carries no provenance block, because a performer does not state its own: CI recomputes the verdicts from the submitted transcript, refuses a mismatch, signs the record with a key the instrument never holds, and writes that block itself.

No credential the run was driven under reaches the branch. The ixit the record carries names environment variables, never values, and the recorded exchanges withhold the credential header.

The identity that opens the pull request is a GitHub App: VEREDICTUM_GITHUB_APP_ID, VEREDICTUM_GITHUB_APP_KEY, VEREDICTUM_GITHUB_INSTALLATION_ID and VEREDICTUM_REGISTRY_REPO. Its installation token is short-lived and revocable, which a signing key on a public host would not be, and it is the only identity permitted to open a console entry. With any of the four unset the step says what to configure and offers no button.

Verify: checking a record without trusting us

/verify is public. It needs no run, no server and no account: upload a bundle, and the published library recomputes every digest the manifest names and checks the detached signature over it.

The upload is a plain HTML form posting to a server route. There is no JavaScript in it, so it works before the client bundle has loaded and works with scripting switched off. Uploaded bundles are transient: they unpack into a scratch directory, are checked, and are swept on a short timer.

A tampered file names itself. Change one byte of one document and the row for that file reads mismatched, with the digest the bytes actually produce beside the digest the manifest promised. Change the manifest instead and the signature is rejected while every remaining digest still matches, which says precisely which of the two claims failed.

Two things are permanent furniture on that page. The first is the honesty box, which renders on every outcome including a clean one: a valid signature proves integrity and origin since signing, and nothing else. It does not prove the conditions the run executed under, that the system under test is what the record says it is, or that the catalogue covered everything the specification defines. The second is the command line that does the same job:

veredictum verify-record --record <dir> --key <public-key>

The manifest’s signature is an ordinary detached OpenPGP signature, so gpg --verify accepts a bundle too. Nobody has to trust this console to check this console.

The key the page checks against is baked into the image at /app/keys/registry-signing.pub.asc, and VEREDICTUM_VERIFY_KEY names it by default, so a fresh instance verifies a published record with no configuration. It is the public half of the key that signs a registry record, which is what a reader needs and all a reader needs. Point that variable at another file to check records signed by another key.

Benchmarks: speed records, read as speed records

/benchmarks reads the JSON document a veredictum bench run writes. It lists every bench-result*.json under the mounted output directory and takes uploaded ones through the same plain-form mechanism /verify uses. An uploaded batch is transient and swept on a timer, because the console stores nothing of its own.

The first thing on every one of these pages is the boundary statement, read verbatim out of the record: a bench result is a comparative speed measurement, and it is not a conformance record, not a certificate, and not a performance-class rating. A table of speed numbers is exactly the artifact somebody quotes out of context, so the sentence that says what it is not travels with it.

One record opens in full. The header names the pack, the system, the machine that offered the load, the seed and the scale, then says whether the record may be offered for ranking and, when it may not, which requirement it misses and what that requirement asks for. The posture block follows, one line per disclosed item, each labelled verified where a black-box canary read it off the running system at both ends of the measured window and declared-only where released ITS-REST discloses nothing to read. Two speed numbers are comparable only when the same features were switched on behind them, which is why the posture reads before any figure.

Then the numbers: the cross-repetition percentiles per phase in microseconds with the millisecond reading beside them, the failed-arrival share of every repetition and phase on the target and on each baseline, the same-machine baselines with their pinned image digests and upstream recipe, and the relative index the run derived against each of them. Every figure carries the discipline that produced it, because a closed-loop average and an open-loop percentile answer different questions and are never read against one another.

Each operation carries the standard HdrHistogram V2 encoding of its own latencies, so every percentile on the page is recomputable from the record itself. The console tabulates rather than draws it: decoding one is the engine’s own histogram reader, which the console reaches once its engine pin carries the bench module.

Selecting two or more records aligns them side by side, one column per record, one row per phase, operation and metric. What makes the view worth having is the block above the numbers: the columns’ packs, generator hosts, posture profiles, full disclosures and scale factors are compared, and every disagreement is stated before a reader reaches a cell. A column that is not submittable says so with the requirement it misses; a set of columns taken on different hosts where one carries no relative index says that nothing in the table is comparable across them.

Running it in public

The hosted instrument at console.veredictum.eu is this console served publicly, and a public console has two problems a local one does not.

It drives whatever endpoint a visitor names. VEREDICTUM_POSTURE=hosted turns on the target guard: loopback, private, link-local, unique-local, unspecified, multicast, carrier-shared and broadcast addresses are refused in both families before any socket opens, and the name is resolved first, because a name under a visitor’s control resolving to a private address is the whole problem. The variable’s default is local, which refuses nothing — driving a CDR at localhost is the normal local case. A value that is neither refuses to start, rather than falling back to the permissive one on a typo.

It has to tell its visitors apart. With no login, that is the peer address. Behind a proxy the peer is the proxy, so VEREDICTUM_CLIENT_IP_HEADER names the header carrying the real client address — and only a header the operator names is read, because an unconditionally trusted X-Forwarded-For would let any visitor claim any identity and defeat the per-visitor caps. Unset, the socket peer is the whole answer.

Both are documented with the rest of the hosted posture in the repository’s deploy/hosted/ directory.

An address the console does not serve

A path outside the route tree answers 404 and renders it: the same chrome, the path that missed named back to you, and two ways on. Nothing was run and nothing was judged, which is what the page says.

Two real servers, side by side

The same catalogue, the same wizard and the same judgement, driven against two live CDRs pulled at their latest published images: FerroEHR’s quickstart and EHRbase’s official pairing. The run behind each column is the EHR-service case family, and every difference below traces to a case id and its citation.

FerroEHR (latest)	EHRbase (latest)

These captures come from the same E2E harness that gates the console (UI_E2E_REAL_SUTS=1 scripts/ui-e2e.sh): the browser drives the real wizard against the real servers, and the book shows what it photographed.

The conformance method

• The oracle is the released specification text
• The catalogue’s shape: one behaviour, one case
• Positive and negative testing carry equal weight
• When a run goes red
• Where the specification does not say
• Verdicts are computed, never asserted
• Coverage is a mandate

A conformance verdict is only worth what its method is worth. This chapter sets the method out.

The oracle is the released specification text

The vendored openEHR specification text is the oracle, and it is never a suspect. Every expectation in the catalogue cites the section it comes from, and the only thing that can refute an expectation is a better reading of that text. Not a maintainer’s confidence, and not what every other CDR happens to do.

Two components are both oracles for a functional case. The Service Model anchors the operation and the naming the cases use; the ITS-REST specification binds that operation to the wire. Where the ITS-REST prose is silent, the released OpenAPI bundle shipped with the same release grounds the expectation, cited as the OpenAPI document and never passed off as prose. It loses on any conflict with the prose.

Some sources look like authority and are not. The official openEHR Conformance component never reached a stable release: its Platform Conformance Test Schedule says which behaviours are worth testing, which is genuinely useful, but it does not settle what the correct answer is, and in places it contradicts a released component. Its Robot test suites are in the same position. Where any of those disagrees with a released component, the released component wins.

The catalogue’s shape: one behaviour, one case

A case core describes one narrow conformance requirement, in the Service Model’s vocabulary, with no status code, header or media type in it. A separate operation binding says how that operation reaches the wire on a given interface specification. That split is what lets the same case be graded against a future wire specification without rewriting the behaviour it tests.

Errors in a case are expressed as kinds, never as codes. The catalogue distinguishes a duplicate EHR from a non-existent EHR from a missing template from a validation failure, because those are different behaviours; the mapping from a kind to a status code lives in the binding, since the same kind maps to different codes on different operations.

Cases stay small on purpose. One behaviour per case means a red row names one defect, which is the difference between a report a vendor can act on and a report they have to reverse-engineer.

Positive and negative testing carry equal weight

A server that accepts everything passes every positive test. That is why the catalogue treats a refusal as a first-class behaviour with its own cases.

Every invalid shape in the corpus is kept as its own entry, marked invalid, with the defect it carries and the specification reference that makes it invalid. Each one has a valid twin. The valid twin proves the server accepts what it must accept; the invalid twin pins the refusal, so a lenient server fails it. Deleting an invalid shape would silently narrow the claim, so it never happens.

The negative surface the catalogue covers includes:

• Content negotiation. An Accept header nothing can satisfy must produce 406; an unsupported payload media type must produce 415.
• Identity and state conflicts. A duplicate identifier, a stale preceding version, a missing version precondition, a delete of something already deleted.
• Content validity. Missing mandatory attributes, empty lists where the Reference Model requires at least one member, structurally wrong documents, and content committed against the wrong template.
• Scope negatives. A format applied where the specification does not define it.
• Authentication. A sweep over the route table checking that every platform route refuses an unauthenticated request.

An invalid payload cannot be authored through a typed model, because a typed model will not construct it. Those fixtures are therefore raw bytes, which is also what makes them catch encoder and decoder bugs that a construct-then-serialize fixture cannot reach.

When a run goes red

A red row is a statement that the specification, the catalogue and the server do not all three agree. Which one is wrong is decided before anything is changed.

Two reflexes are both refused, and the second is why this instrument exists separately from any server:

1. “The catalogue must be wrong, the server is right.” This is the reflex a vendor brings to a failing conformance run. The catalogue answers it by construction: an expectation traces to a citation, so it is refuted by a better reading of the released text and by nothing else.
2. “The server must be wrong, the instrument is right.” Veredictum is the thing people are being asked to trust, so it is a suspect on every red row, ahead of the server. The first live triage attributed 7 of 7 diagnosed defects to the instrument and none to the server under test.

Each red row is attributed to exactly one of three suspects:

The derivation per row is the same every time: read what was actually sent and received, read what the case and its binding claim to expect and why, then read the governing released specification text first-hand and derive independently what a conformant server must return for the exchange that actually happened. The three-way comparison names the defect.

Two rules keep that honest. A transport fault or a step that could not be resolved classifies as inconclusive on the instrument’s side, never as a server failure. And an attribution naming the server carries the reproduced exchange plus the citation with the decisive sentence quoted, because a verdict about somebody else’s product is held to a higher evidence bar than a verdict about our own.

Warning

A catalogue expectation is never adjusted to match observed server behaviour. A server response is evidence in the three-way comparison. It is not the reference, and treating it as one is how a conformance suite quietly becomes a description of whatever the last server did.

Where the specification does not say

Sometimes the released text leaves a behaviour undefined, or two released documents disagree. Inventing an expectation there would fail every conformant server and teach nobody anything. Hiding it would be worse.

Instead, each such case is recorded in the ambiguity register with four things: the ambiguity, the first-hand citation that establishes it, the handling a runner must apply, and a typed disposition the pipeline branches on.

The dispositions are a closed set:

Four properties make this a transparency mechanism rather than an excuse list:

• An entry has to be proven. A claimed ambiguity that the specification actually defines is a catalogue defect. The entry is removed and the case becomes gating.
• Nothing is absorbed. A report_only or editorial entry must carry a link to an upstream report, and that requirement is enforced by the schema rather than by review. A carried divergence always has an outbound report attached.
• Every case still runs. The register is not an exclusion list. It governs how a spec-silent expectation is derived and whether a genuinely open question gates a certificate.
• A private resolution is non-conformant. A harness that quietly decides an ambiguity for itself is not implementing this schedule.

report_only is a cited, upstream-linked suspension, and it reverts to gating the moment the upstream question resolves. It is not a way to make red rows disappear.

Verdicts are computed, never asserted

A verdict is a pure function of the party’s statement, the recorded results, the catalogue and the capability matrix. The recording step and the judging step are separate commands, so the record survives the judgement and anyone holding the same files can re-derive the same documents.

Measured performance works the same way. A class verdict is re-derived from the HDR histograms embedded in the record, so the stored summary is tamper-checked rather than trusted, and latency is measured from the planned arrival instant under open-loop offered load, which is what stops coordinated omission from hiding a stall.

Coverage is a mandate

A green run over a thin catalogue proves nothing, so coverage is machine-checked rather than asserted. A gate enumerates the wire surface from the released sources alone, the Service Model’s platform interfaces crossed with their ITS-REST branches, and fails on any operation, status-code branch, header rule, negotiation variant or error family that has neither a covering case nor a cited exception.

A behaviour the specification defines and the catalogue misses is a gap to close or an honest boundary recorded in the register. Cases are added; they are never removed to make a run go green.

Authoring the catalogue

• The artifact families
• A case core
• Operation bindings
• The closed vocabularies
• The corpus, and why invalid fixtures stay
• The rules an author works under
• Checking your work

The catalogue is data, not code. You can read it, diff it, and add to it without touching the runner, and a harness written in another language can execute the same files. This chapter is an overview of its shape and the rules an author works under.

The full grammar is the published JSON Schema set, in schemas/. Those files are emitted by veredictum emit-schemas and drift-tested against it, so the published format and the code that reads it are one thing. Author against them; this page will not repeat every field.

The artifact families

Everything lives under one artifact root, which is artifacts/ in the repository.

Chapters under schedule/ follow the openEHR service components: ehr, composition, contribution, directory, query, the three definition_* chapters, demographic, admin, messaging, security, smart, simplified_formats, system and content. A separate performance family holds the measured-workload journey definitions, which are not case cores and do not carry capabilities.

A case core

One file, one behaviour. The fields that shape every case:

• id is a global identifier and is never reused. A retired case keeps its identifier with status: retired.
• test_purpose is the one narrow conformance requirement the case exists to check, in prose. If you cannot write it as one requirement, the case is really two cases.
• spec_refs are the citations, by component, document and section. They are resolved against the vendored specification tree by validate, so a citation that does not resolve is a finding rather than a comment.
• capabilities are the capability names whose verdict this case bears. Keep the list minimal: a failure marks every capability listed. Capabilities the case merely touches go in exercises, which is informative and bears no verdict.
• requires is typed prerequisite state, not prose. An empty server, a provisioned template, an EHR with no commits, a folder tree. Each provisioned object mints a named handle the flow can reference.
• flow is the ordered steps of a functional case: which operation, with what arguments, what outcome kind is expected, and what to capture from the response for a later step to use.
• parameters is the data-set dimension. A test is one case run against one data set, so a value matrix in a case produces one test per row, with preconditions re-established around each row by default.
• postconditions are typed assertions evaluated after the flow.

Two more fields carry the honesty:

• applies states the specification version range a case is applicable in. Applicability is decided by the typed fields only: applies, status, capabilities, option, and the requires block. A case selection puts out of scope is recorded not-applicable with its citation, never a silent skip.
• guards is cited prose about the case: where the expectation was authored from, what the row does not claim, the adjudication it rests on, the assumptions it makes. It selects nothing, and validate refuses a guard phrased as if it did.
• ambiguities lists the register entries the case is subject to.

What a case core deliberately does not contain: any status code, header name, or media type. Those live in the binding.

Operation bindings

A binding is the wire layer for one Service-Model operation on one interface specification. It maps each outcome kind the case can expect to the wire result that realizes it, and it says what to capture from the response.

The mapping is per operation, because the same kind lands differently in different places. A validation failure is one status code on a composition operation and a different one on EHR creation, and the released specification is what decides which. A kind a binding cannot map is a validation finding.

There are 249 bindings today, and validate checks that every operation a case calls has one.

The closed vocabularies

Three of the vocabularies are closed enumerations, and that is what makes the catalogue machine-checkable.

• Outcome kinds. A case says created, not_found, precondition_failed, validation_failed and so on. There are 26 of them, extensible only by a schedule release, and a case that speaks anything else fails validation. Codes never appear in a case.
• Selectors. How an assertion addresses part of a response body or a header, including the ignore-sets that let a case compare a document while ignoring the fields a server legitimately assigns.
• The capability matrix. The capability, family and profile-tier structure the verdict machinery computes over. This is the openEHR Platform Profiles book’s capability tables as data.

The wire surface under vocab/ is the fourth machine-readable list, and it is the input to the coverage gate: the operations, status-code branches, header rules, negotiation variants and error families the released sources define. A behaviour in that list with no covering case and no cited exception fails the gate.

The corpus, and why invalid fixtures stay

A corpus entry carries its own adjudicated verdict: valid, or invalid with the defect it carries and the specification reference that makes it invalid. Every invalid shape has a valid twin and is authored as raw bytes, for the reasons the conformance method sets out. Removing an invalid fixture narrows the claim without changing any visible count, which is why it does not happen.

Vendored breadth packs carry the same discipline. A pack is exercised in full, with any skip adjudicated and recorded, so a pack never sits in the tree implying coverage it does not have.

The rules an author works under

1. Cite the released specification. An expectation without a resolving citation is a finding. The openEHR Conformance component’s test schedule is a guide to which behaviours are worth covering, and it is not authority for what the correct answer is.
2. One behaviour per case. If a failure of your case could mean two different defects, split it.
3. Never encode a status code in a case core. That is the binding’s job, and keeping the split intact is what lets the behaviour outlive the wire specification.
4. Add cases; never remove one to go green. A removal that makes a run pass is a narrowed claim, not a fix.
5. Never adjust an expectation to match an observed response. Change it only with a new citation that says the old reading was wrong.
6. Record a specification silence, do not resolve it. It goes in the ambiguity register with a typed disposition and an upstream report, and a private resolution makes a harness non-conformant.
7. Keep the invalid twin. See above.

Checking your work

veredictum validate --root artifacts --specs specs/openehr

Zero findings is the only passing result, and the gates behind that line are in the command reference.

Run it before you drive anything against a server. A catalogue defect found by a server run costs a great deal more to diagnose than the same defect found by validate in a second.

Glossary

• The parties and the claim
• The catalogue and the run
• The verdict
• The authorities

The conformance vocabulary comes from two places: the ISO conformance-testing tradition (ICS, IXIT, SDoC) and the openEHR specifications themselves (RM, AQL, the component names). This page defines every term the instrument and the console put in front of you.

The parties and the claim

• Party: the vendor whose product is being graded and who makes the conformance claim. The two documents below are theirs to write and travel with the submission they describe; this repository commits neither.
• ICS (Implementation Conformance Statement): the party’s claim document, statement.json. It declares which profile tiers and which capabilities the product claims. A verdict certifies this claim against the recorded evidence, so a run without a statement produces results but no verdict.
• ICS proforma: the blank form an ICS fills in, artifacts/vocab/capability_matrix.yaml. ISO/IEC 9646-7 assigns every cell of it to the test specification’s author except the support and supported-values columns, which belong to the supplier of the implementation. So the proforma is published here, one row per capability with its specification citation, and the answers never are. The term comes from the ISO/IEC 9646 conformance-testing methodology.
• IXIT (Implementation eXtra Information for Testing): the deployment facts needed to drive the claim, ixit.json: the endpoint URLs, the authentication mode, the names of the credential environment variables. Same ISO/IEC 9646 origin. The console writes one for you from the Connect form.
• SDoC (Supplier’s Declaration of Conformity): the self-declaration half of the statement, in the sense of ISO/IEC 17050: the supplier declares, the instrument checks, and nobody grades their own homework.
• SUT (System Under Test): the running server a campaign drives, here always a CDR reached over its own REST wire.
• CDR (Clinical Data Repository): a server that stores and serves openEHR clinical data, such as EHRbase or FerroEHR.

The catalogue and the run

• Catalogue: the machine-readable test schedule under artifacts/: case cores, operation bindings, vocabulary, corpora, and the ambiguity register. validate holds it to zero findings before any server is composed.
• Case: one spec-cited behaviour, one file, one row in the record. One behaviour per case is a design rule: a red row then names exactly one defect.
• Case id: the stable identifier of a case, interface first, such as I_EHR_SERVICE.create_ehr-main.
• Case-id filter: a plain substring match over case ids, the engine’s --filter flag. Every case whose id contains the typed text is in scope; an empty filter means the whole catalogue. I_EHR_SERVICE. selects the EHR-service family, create_ehr-main selects one case. The filter narrows the run, never the claim: a claimed capability whose cases were filtered out is reported not_evidenced, so a narrow run cannot pose as full coverage.
• Corpus: the committed wire payloads the cases post, valid and invalid. An invalid entry is deliberate: it pins a refusal the spec requires, so a lenient server fails it.
• Excused: a case the run did not hold against the server, with the citation that permits the excuse: an out-of-claim capability, or an instance the party does not declare. Excuses are printed, never silent.
• Ambiguity register: the record of behaviours where the released specifications are genuinely silent or contradictory, each with a typed disposition and, where one exists, its upstream report. Silence goes here; it is never resolved privately.

The verdict

• Verdict: the computed answer to the claim, produced by a pure function over the statement, the record, the catalogue and the capability matrix. It is computed, never asserted, and re-running it over the same record produces the same bytes.
• Profile tier: one of the CNF profile levels a party can claim: CORE, STANDARD, OPTIONS, and the SEC-BASIC security family. A tier passes only when every capability it requires is passed.
• Capability: one named unit of claimable behaviour, such as EhrOperations or QueryProvisioning. The capability matrix maps each tier to the capabilities it requires, and each capability to the cases that evidence it.
• Evidence tokens: the per-capability answer in the matrix. passed (every selected gating case passed), failed (one failed), inconclusive (one errored and none failed; never counted against the server, but blocking green), not_evidenced (claimed, and no case produced a gating pass or fail), not_claimed (absent from the party’s ICS).
• Errored (inconclusive): a row whose exchange could not be judged: a status mapping to no declared outcome, a transport fault. By the attribution law this is never a server failure; it is a defect in the runner or the catalogue until adjudicated.

The authorities

• Oracle: the released openEHR specification text vendored under specs/. It is the only authority an expectation may cite, and it is never a suspect when a run goes red.
• CNF: openEHR’s conformance specification component. Its Platform Conformance Test Schedule names which behaviours to cover; the released components say what the correct answer is.
• Spec components: RM (the Reference Model: the data structures), AQL (the Archetype Query Language), AM (the Archetype Model), BASE (foundation types and identifiers), TERM (terminology), SM (the Service Model: the abstract operations), ITS (the Implementation Technology Specifications for REST, JSON and XML: how the operations land on a wire).
• Attribution law: the discipline applied to every red row before anything changes. The failure belongs to exactly one of three suspects: the server, the runner, or the catalogue. Which one is decided by comparing spec-required against catalogue-expected against observed, with the cited text as the reference.