Running the instrument

• 1. Check the catalogue
• 2. Declare your deployment
• 3. Drive the catalogue
• 4. Compute the verdicts
• Measured performance and stress
• Where to look when something goes wrong

A conformance campaign is four steps. Each writes a file the next one reads, so you can stop after any of them, inspect what was produced, and resume.

1. Check the catalogue

Before a server is involved, check the catalogue you are about to grade with:

veredictum validate --root artifacts --specs specs/openehr

This is every machine check over the artifact tree: identifier uniqueness, citation resolution against the vendored specification text, binding completeness, coverage of the enumerated wire surface, and claim completeness against the committed party statements. Zero findings is the only passing result, and the command exits 1 when there is even one.

Pass --specs every time. Without it the citation and Service-Model resolution gates do not run, and the case count still prints, which looks like a pass over a catalogue that was never fully checked.

2. Declare your deployment

The instrument needs to know where your server is and how to authenticate to it. That declaration is the IXIT file, and copying an example is the fastest way to a correct one:

cp -r party/ehrbase party/mine

The directory holds two files, and the split between them matters:

• ixit.json describes the deployment. Endpoints per instance, the authentication mode, and for a measured run an environment block naming the hardware and topology the numbers were produced on. Credentials are named, never carried: the file holds the names of the environment variables the instrument reads the user and password from, so no secret ever enters an artifact you might publish.
• statement.json is your declaration of claims. It names the product and version, the specification versions it targets, and the capabilities it claims to implement. The verdict machinery reads it as the thing being tested against the record.

A typical ixit.json declares three instances, because a full run needs to speak to the server as three different callers: an ordinary clinical user, an administrator, and no one at all. The unauthenticated instance is what lets the security cases check that a route refuses an anonymous request.

3. Drive the catalogue

veredictum run --root artifacts --ixit party/mine/ixit.json --out out/ \
    --sut-name my-cdr --sut-version 1.2.3 --statement party/mine/statement.json

The command drives every applicable case against your endpoints and writes out/results.json, a record of what was sent, what came back, and how it was classified. It also writes out/run-exceptions.json, which lists the cases the interpreter could not drive at all.

results.json is a record, not a judgement. That separation is the point: the recorded exchange stays available for anyone to re-read, and the judgement is computed from it in a separate step that touches no network.

Passing --statement here changes what runs. A case gated on an option your statement does not declare is recorded as not-applicable at drive time rather than driven, which is the test-selection discipline ISO/IEC 9646 describes. Omit the flag and everything is driven, which is what you want when you are exploring an unfamiliar server rather than grading a declared one.

--filter takes a substring matched against case identifiers, which is how you re-drive one chapter while working on a fix. The resulting results.json holds only the cases that ran, so a filtered run is a working tool and never the record you submit.

The command exits 1 if any case failed or errored, so a shell script can gate on it.

Note

A functional run never re-measures. If a results.json already exists at the --out path, its measurement records are carried forward, so running the functional catalogue again after a measured run does not discard the performance evidence. A file that is present but unreadable stops the run instead, because carrying zero measurements past it would drop that evidence silently.

4. Compute the verdicts

veredictum verdicts --root artifacts --statement party/mine/statement.json \
    --results out/results.json --out out/

This step is a pure function of the statement, the recorded results, the catalogue and the catalogue’s capability matrix. Run it twice on the same files, on any machine, and you get the same bytes out. It writes:

Nothing in these documents is asserted by hand. A number that appears in them was computed from the record in the same run that printed it.

Measured performance and stress

Those four steps cover functional conformance. Two more instruments produce the other kinds of evidence, and both need the environment block in your IXIT file filled in, because a throughput number without the deployment described says nothing.

• veredictum perf earns a volumetric class (POC, S, L or R) with an open-loop measured run and merges the measurement record into an existing results.json. The normative window is one hour; --hours extends it to 2, 4, 6, 8 or 12, which is a stricter demonstration. Nothing shorter than the case exists.
• veredictum stress climbs geometric load steps to find where the deployment breaks and writes stress.json. It is exploration only and is never a conformance record, which is why it is class-free by design.

Both want an idle machine and a deployment whose resource limits match the envelope you are claiming. A measured run on a laptop that is also running a browser measures the browser.

Where to look when something goes wrong

A red row is not presumptive evidence of a bug in your server. It is evidence that the specification, the catalogue and the server do not all three agree, and which one is wrong is a question that gets answered before anything is changed. The conformance method sets out how, and what evidence an attribution has to carry.

The command reference lists every subcommand with its real flags.