Running the instrument

• 1. Check the catalogue
• 2. Declare your deployment
• 3. Review your claim
• 4. Drive the catalogue
• 5. Compute the verdicts
• Measured performance, stress and speed
• Where to look when something goes wrong

A conformance campaign is four steps. Each writes a file the next one reads, so you can stop after any of them, inspect what was produced, and resume.

1. Check the catalogue

Before a server is involved, check the catalogue you are about to grade with:

veredictum validate --root artifacts --specs specs/openehr

Zero findings is the only passing result, and the command exits 1 when there is even one. The command reference lists the gates it runs.

Pass --specs every time. Without it the citation and Service-Model resolution gates do not run, and the case count still prints, which looks like a pass over a catalogue that was never fully checked.

2. Declare your deployment

The instrument needs to know where your server is and how to authenticate to it. That declaration is the IXIT file, and it is yours to write, against schemas/ixit.schema.json. Put it in a directory of your own beside the clone — mine/ixit.json in the commands below — with your claim next to it. fixtures/declaration/ carries both shapes filled in for a product that does not exist, which is a template to read and never a claim to copy.

This repository commits no declaration about anybody’s product. ISO/IEC 9646-7 splits an ICS proforma in one place: every cell belongs to the proforma specifier except the support and supported-values columns, which belong to the supplier of the implementation. The proforma is artifacts/vocab/capability_matrix.yaml, published here with a specification citation per row. The answers are yours.

The two files, and the split between them matters:

• ixit.json describes the deployment. Endpoints per instance, the authentication mode, and for a measured run an environment block naming the hardware and topology the numbers were produced on. Credentials are named, never carried: the file holds the names of the environment variables the instrument reads the user and password from, so no secret ever enters an artifact you might publish.
• statement.json is your declaration of claims. It names the product and version, the specification versions it targets, and the capabilities it claims to implement. The verdict machinery reads it as the thing being tested against the record.

A typical ixit.json declares three instances, because a full run needs to speak to the server as three different callers: an ordinary clinical user, an administrator, and no one at all. The unauthenticated instance is what lets the security cases check that a route refuses an anonymous request.

3. Review your claim

veredictum validate --root artifacts --specs specs/openehr \
    --statement mine/statement.json

--statement adds the static conformance review of ISO/IEC 9646-1 and -7 to the same command that checks the catalogue. It reports a capability you claim that the catalogue holds no verdict-bearing case for, a Signing claim your ixit declares no posture for, and a served-extension family the catalogue’s wire surface does not carry. Each of those would otherwise become a row that can never carry evidence, discovered mid-run.

4. Drive the catalogue

veredictum run --root artifacts --ixit mine/ixit.json --out out/ \
    --sut-name my-cdr --sut-version 1.2.3 --statement mine/statement.json

The command drives every applicable case against your endpoints and writes out/results.json, a record of what was sent, what came back, and how it was classified. It also writes out/run-exceptions.json, which lists the cases the interpreter could not drive at all.

results.json is a record, not a judgement. That separation is the point: the recorded exchange stays available for anyone to re-read, and the judgement is computed from it in a separate step that touches no network.

Passing --statement here changes what runs. A case gated on an option your statement does not declare is recorded as not-applicable at drive time rather than driven, which is the test-selection discipline ISO/IEC 9646 describes. Omit the flag and everything is driven, which is what you want when you are exploring an unfamiliar server rather than grading a declared one.

--filter takes a substring matched against case identifiers, which is how you re-drive one chapter while working on a fix. The resulting results.json holds only the cases that ran, so a filtered run is a working tool and never the record you submit.

The command exits 1 if any case failed or errored, so a shell script can gate on it.

Note

A functional run never re-measures. If a results.json already exists at the --out path, its measurement records are carried forward, so running the functional catalogue again after a measured run does not discard the performance evidence. A file that is present but unreadable stops the run instead, because carrying zero measurements past it would drop that evidence silently.

5. Compute the verdicts

veredictum verdicts --root artifacts --statement mine/statement.json \
    --results out/results.json --out out/

This step is a pure function of the statement, the recorded results, the catalogue and the catalogue’s capability matrix. Run it twice on the same files, on any machine, and you get the same bytes out. It writes:

Nothing in these documents is asserted by hand. A number that appears in them was computed from the record in the same run that printed it.

Measured performance, stress and speed

Those four steps cover functional conformance. Three more instruments produce the other kinds of evidence.

• veredictum perf earns a volumetric class (POC, S, L or R) with an open-loop measured run and merges the measurement record into an existing results.json. The normative window is one hour; --hours extends it to 2, 4, 6, 8 or 12, which is a stricter demonstration. Nothing shorter than the case exists.
• veredictum stress climbs geometric load steps to find where the deployment breaks and writes stress.json. It is exploration only and is never a conformance record, which is why it is class-free by design.
• veredictum bench measures comparative speed against any reachable CDR and writes a bench-result document. It needs no artifact root, no IXIT and no statement, and it is a speed record and never a conformance record. The benchmark board ranks the submitted ones.

perf and stress both need the environment block in your IXIT file filled in, because a throughput number without the deployment described says nothing. All three want an idle machine and a deployment whose resource limits match the envelope you are claiming. A measured run on a laptop that is also running a browser measures the browser.

Where to look when something goes wrong

A red row is not presumptive evidence of a bug in your server. It is evidence that the specification, the catalogue and the server do not all three agree, and which one is wrong is a question that gets answered before anything is changed. The conformance method sets out how, and what evidence an attribution has to carry.

The command reference lists every subcommand with its real flags.