Veredictum
Benchmark methodology

What the benchmark actually creates, and what it actually measures

The benchmark board ranks openEHR clinical data repositories by speed. This page says what produced those numbers: the data each benchmark pack creates, the requests it then issues, how long it issues them for, and what each request is there to find out. It is written for someone who has never run the tool.

Nothing below is typed by hand. A pack is versioned data compiled into the veredictum binary, and the binary emits its own description as bench-packs.json, which this page is rendered from. A change to a pack that left this page saying something else would fail the build.

A bench number is not a conformance verdict. This is a benchmark record for comparative speed. It is not a conformance record, not a certificate, and not a performance-class rating; a bench result may motivate a class run, never substitute for one. Conformance is a separate record, produced by a different part of the same tool: it drives a catalogue of spec-cited cases and judges each answer against the released openEHR specifications. Speed says nothing about whether a server is correct, and nothing on this page claims otherwise.

How a run works

Seed once, measure several times

Every pack follows the same shape, and each term below means the same thing further down this page, on the board, and inside every published record.

What gets created. A run starts by uploading the pack's operational template, creating EHRs, and committing compositions into them, all through the same public REST API a real client would use. The tool reaches no database and reads no server-internal state. The population that load leaves behind is what every measured request then addresses, so two runs of the same pack measure work against the same amount of data.

Closed-loop. The next request goes out once the previous one has come back. A slow server therefore receives fewer requests, which is why a closed-loop phase reports throughput and a whole-loop average rather than percentiles. It is the discipline a single-client harness uses, and it is how the bulk load and the read walk are reported.

Open-loop. Arrivals fire at instants planned before the phase starts, at a rate the pack version pins, whether or not earlier requests have answered. Every latency is measured from the planned instant rather than from the moment the request went out, so a server that stalls for a second charges that second to every arrival that was due during it. This is where the percentiles come from, and it is what stops a stall from hiding behind a lower request count.

Warmup. The first seconds of an open-loop phase are dispatched and then thrown away. A cold cache and a just-started process describe the first minute of a deployment rather than the deployment, so those arrivals are made and their latencies are discarded. Each pack below states how long its warmup is and how many arrivals it costs.

The seed. Every pack declares one seed, and every draw a run makes derives from it: which operation each arrival offers, which EHR or composition it addresses, and the query parameters a measured AQL class substitutes. Two runs of the same pack version at the same scale therefore offer the same work in the same order, on any machine and against any system, which is what makes two records comparable at all. The seed is a property of the pack rather than of the run, so it is not an operator input and no command-line flag moves it.

The posture. Two speed numbers are comparable only when the same features were switched on behind them, so every pack defines named posture profiles and a run declares exactly one with --posture. The record then carries one line per disclosed item, and each line is labelled verified or declared-only: a black-box canary reads the item off the running system before and after the measured window, and where released ITS-REST surfaces nothing to read the declaration is carried as a claim and says so. A canary that contradicts the declaration, and a pair of brackets that disagree with each other, both refuse the whole run. Authentication and TLS are facts of the invocation rather than choices of the profile, so a profile leaves them to the run. Each pack below lists the profiles it defines and what each one declares, and every record names the profile its own run was measured under.

Repetitions. The population is built once and the measured phases are then repeated. Every figure a record publishes is the median across repetitions, with the spread beside it, because one repetition measures a moment.

The relative index. The relative index is the target's cross-repetition median divided by the baseline's cross-repetition median for the same phase, operation and metric, both measured on the same host in the same session. It is dimensionless. On a latency metric a value below 1.0 means the target answered faster than the baseline and above 1.0 slower; on throughput the sense inverts, because there a larger number is the faster system. A record earns one by composing the pinned reference deployments on the same machine, in the same session, and driving the same pack at the same seed against them. That is the figure the board sorts on, because milliseconds taken on somebody else's hardware cannot be compared with yours.

The packs

3 packs, each pinned by version

Two records are comparable when they name the same pack at the same version. A change to any figure below is a change to the work, so it moves the version and the older records stop being comparable with the newer ones.

aql-mix version 1.0.0

seed 4706627057783240704 · 2 phases · failed-arrival ceiling 0.01 · drive it with --pack aql-mix

seed closed-loop, runs until it is finished queries warmup 15s, discarded · measured 60s at 24 arrivals/s

The seed phase builds the population. It creates 50 EHRs through the public API and commits the same composition 20 times into each, on 8 workers, leaving 1,000 compositions behind. Every later phase reads and writes against exactly that population. The phase is closed-loop: the next request goes out once the previous one has answered, so it reports bulk-load throughput and never a latency claim.

The queries phase offers arrivals on a fixed schedule: 24 a second for 60 seconds, after a 15-second warmup whose 360 arrivals are dispatched and then discarded. 1,440 arrivals are measured. They fire at their planned instants whether or not an earlier request has come back, and every latency is measured from the planned instant, so a server that stalls shows the stall in its percentiles instead of quietly receiving fewer requests.

What the queries phase offers, and why each one is in the mix
OperationRequestShareOfferedWhat it probes
adhoc_query_point_lookupPOST /query/aql14/sthe indexed-read floor: one composition addressed by its own uid inside one EHR, the cheapest query a server can answer
adhoc_query_ehr_scanPOST /query/aql14/sthe loaded-database shape: every composition in one EHR projected by uid, so the cost follows how much that EHR holds
adhoc_query_filteredPOST /query/aql14/sthe value index: a systolic magnitude threshold over the observation leaves of one EHR, with the threshold drawn per arrival so no result set can be memoized
adhoc_query_populationPOST /query/aql14/sthe cross-EHR planner: the same magnitude threshold with no EHR scope and a fetch bound, so the server picks an access path over the whole population
adhoc_query_aggregatePOST /query/aql14/sthe columnar shape: one COUNT over the population that threshold matches, which returns a single row and reads every value behind it
adhoc_query_ordered_pagePOST /query/aql14/ssorting and pagination: an ORDER BY over composition start time read through a moving fetch window, the shape a paged user interface issues

A run against this pack declares one posture profile, which says what was switched on behind the numbers. minimal is what a run takes when it names none.

The posture profiles aql-mix defines, and what each one declares
ProfileWhat it switches onDeclares
minimal defaultThe bare spec-conformant surface: no audit trail, unsigned versions, commits validated against the operational template, uncompressed responses, one tenant.audit off
commit_validation template
compression off
tenancy single
version_signing none
The bytes it offers, and where they came from

vital_signs.opt · operational template · 217.9 KiB · application/xml

The openEHR Clinical Knowledge Manager's own Operational Template export for template id 'Vital signs' (CKM cid 1013.26.380, <https://ckm.openehr.org/ckm>), vendored byte-identically and rooted at openEHR-EHR-COMPOSITION.encounter.v1.

sha256 3a0d31bd3b5dc6329e53c0d6f22fdbaece62c684136b86139d0729cff8796128

vital_signs_composition.json · composition · 22.1 KiB · application/json

The composition attached to post 8 of the openEHR community's vital-signs benchmark thread (<https://discourse.openehr.org/t/17224>), vendored byte-identically: eight OBSERVATION entries under openEHR-EHR-COMPOSITION.encounter.v1, rm_version 1.0.2, declaring template id 'Vital signs'.

sha256 468081c259c737d35d7f80403562b3f333e479d267286faf80fd7c087eaba947

vital_signs_composition.missing_composer.json · invalid composition · 21.8 KiB · application/json

Derived in this repository from vital_signs_composition.json by deleting the mandatory COMPOSITION.composer member and nothing else, so a server that validates a commit against the reference model refuses it.

sha256 f0598db5ab447b371ead28cba0f841f72370dbbf93db98d5b8e477910a42688d

Each digest is verified when the pack loads, so a run refuses to start if a single byte of any fixture has moved. Two records that name the same pack version therefore offered the same bytes to both systems. A fixture marked invalid composition is never committed by a phase and never enters the measured population: the commit-validation canary offers it once before and once after the measured window, to see whether the server refuses it as the declared posture says it should.

The pack's own description, as every record carries it

Measures AQL query speed over the same Vital signs population the community-vitals pack seeds, so a query figure and a read figure describe the same corpus. The seed phase creates 50 EHRs and commits the same composition 20 times into each, on a pool of 8 workers. This pack version pins that population, and it is sized for query shapes: large enough that a query has to choose an access path, small enough to load before a measured window opens. The measured phase is open-loop at 24 arrivals a second for 60s after a 15s warmup, over six query classes at equal share, so each class is offered at 4 arrivals a second and every class returns the same number of samples. Each class posts one AQL statement to /query/aql, accepts only 200, and counts every other answer in its own error class, so a server that refuses one shape never contaminates another class's percentiles. Every query parameter draws from the run's seeded streams: the systolic threshold, the page offset, and the EHR or composition each arrival addresses, so no arrival repeats the previous one's result set and the whole draw is reproducible from the seed the record discloses. The six classes: adhoc_query_point_lookup probes the indexed-read floor: one composition addressed by its own uid inside one EHR, the cheapest query a server can answer; adhoc_query_ehr_scan probes the loaded-database shape: every composition in one EHR projected by uid, so the cost follows how much that EHR holds; adhoc_query_filtered probes the value index: a systolic magnitude threshold over the observation leaves of one EHR, with the threshold drawn per arrival so no result set can be memoized; adhoc_query_population probes the cross-EHR planner: the same magnitude threshold with no EHR scope and a fetch bound, so the server picks an access path over the whole population; adhoc_query_aggregate probes the columnar shape: one COUNT over the population that threshold matches, which returns a single row and reads every value behind it; adhoc_query_ordered_page probes sorting and pagination: an ORDER BY over composition start time read through a moving fetch window, the shape a paged user interface issues. Fixture provenance: the operational template is the vendored CKM export for template id 'Vital signs' (CKM cid 1013.26.380) and the composition is the attachment on post 8 of <https://discourse.openehr.org/t/17224>, both byte-identical and pinned by sha256. This pack version pins a failed-arrival ceiling of 0.01: a record in which any repetition, phase and operation loses a larger share of its arrivals, on the target or on any baseline, is not submittable, because percentiles taken over failed arrivals measure the failure rather than the system.

community-vitals version 1.0.0

seed 4859222839474415988 · 3 phases · failed-arrival ceiling 0.01 · drive it with --pack community-vitals

write closed-loop, runs until it is finished read_walk closed-loop, runs until it is finished read_open_loop warmup 15s, discarded · measured 60s at 200 arrivals/s

The write phase builds the population. It creates 100 EHRs through the public API and commits the same composition 1,000 times into each, on 1 worker, leaving 100,000 compositions behind. Every later phase reads and writes against exactly that population. The phase is closed-loop: the next request goes out once the previous one has answered, so it reports bulk-load throughput and never a latency claim.

The read_walk phase walks the whole population in order, issuing 7 reads against every committed composition on 1 worker. That is what a single-client harness does, and it is closed-loop by construction, so the figure it reports is the whole-loop average per request. It is the number that compares with a published harness figure, and it is not a percentile.

The 7 reads the read_walk phase issues against every composition, in this order
#OperationRequest
1get_composition_latestGET /ehr/{ehr_id}/composition/{uid}
2get_composition_at_timeGET /ehr/{ehr_id}/composition/{uid}?version_at_time={at_time}
3get_versioned_compositionGET /ehr/{ehr_id}/versioned_composition/{uid}
4get_versioned_composition_version_latestGET /ehr/{ehr_id}/versioned_composition/{uid}/version
5get_versioned_composition_version_at_timeGET /ehr/{ehr_id}/versioned_composition/{uid}/version?version_at_time={at_time}
6get_versioned_composition_version_by_idGET /ehr/{ehr_id}/versioned_composition/{uid}/version/{version_uid}
7get_versioned_composition_revision_historyGET /ehr/{ehr_id}/versioned_composition/{uid}/revision_history

The read_open_loop phase offers arrivals on a fixed schedule: 200 a second for 60 seconds, after a 15-second warmup whose 3,000 arrivals are dispatched and then discarded. 12,000 arrivals are measured. They fire at their planned instants whether or not an earlier request has come back, and every latency is measured from the planned instant, so a server that stalls shows the stall in its percentiles instead of quietly receiving fewer requests.

What the read_open_loop phase offers, and why each one is in the mix
OperationRequestShareOfferedWhat it probes
get_composition_latestGET /ehr/{ehr_id}/composition/{uid}128.57/sthe harness's latest-version composition read, the read a client issues most
get_composition_at_timeGET /ehr/{ehr_id}/composition/{uid}?version_at_time={at_time}128.57/sthe harness's composition read at an instant, which resolves a version by time
get_versioned_compositionGET /ehr/{ehr_id}/versioned_composition/{uid}128.57/sthe harness's read of the VERSIONED_COMPOSITION container itself
get_versioned_composition_version_latestGET /ehr/{ehr_id}/versioned_composition/{uid}/version128.57/sthe harness's latest-version read through the versioned object
get_versioned_composition_version_at_timeGET /ehr/{ehr_id}/versioned_composition/{uid}/version?version_at_time={at_time}128.57/sthe harness's version-at-an-instant read through the versioned object
get_versioned_composition_version_by_idGET /ehr/{ehr_id}/versioned_composition/{uid}/version/{version_uid}128.57/sthe harness's read of one version by its own identifier
get_versioned_composition_revision_historyGET /ehr/{ehr_id}/versioned_composition/{uid}/revision_history128.57/sthe harness's revision-history read, which walks every version of the object

A run against this pack declares one posture profile, which says what was switched on behind the numbers. minimal is what a run takes when it names none.

The posture profiles community-vitals defines, and what each one declares
ProfileWhat it switches onDeclares
minimal defaultThe bare spec-conformant surface: no audit trail, unsigned versions, commits validated against the operational template, uncompressed responses, one tenant.audit off
commit_validation template
compression off
tenancy single
version_signing none
clinical-defaultA clinical deployment's usual configuration: an audit trail written to the deployment's own store, unsigned versions, commits validated against the operational template, uncompressed responses, one tenant.audit internal
commit_validation template
compression off
tenancy single
version_signing none
The bytes it offers, and where they came from

vital_signs.opt · operational template · 217.9 KiB · application/xml

The openEHR Clinical Knowledge Manager's own Operational Template export for template id 'Vital signs' (CKM cid 1013.26.380, <https://ckm.openehr.org/ckm>), vendored byte-identically and rooted at openEHR-EHR-COMPOSITION.encounter.v1.

sha256 3a0d31bd3b5dc6329e53c0d6f22fdbaece62c684136b86139d0729cff8796128

vital_signs_composition.json · composition · 22.1 KiB · application/json

The composition attached to post 8 of the openEHR community's vital-signs benchmark thread (<https://discourse.openehr.org/t/17224>), vendored byte-identically: eight OBSERVATION entries under openEHR-EHR-COMPOSITION.encounter.v1, rm_version 1.0.2, declaring template id 'Vital signs'.

sha256 468081c259c737d35d7f80403562b3f333e479d267286faf80fd7c087eaba947

vital_signs_composition.missing_composer.json · invalid composition · 21.8 KiB · application/json

Derived in this repository from vital_signs_composition.json by deleting the mandatory COMPOSITION.composer member and nothing else, so a server that validates a commit against the reference model refuses it.

sha256 f0598db5ab447b371ead28cba0f841f72370dbbf93db98d5b8e477910a42688d

Each digest is verified when the pack loads, so a run refuses to start if a single byte of any fixture has moved. Two records that name the same pack version therefore offered the same bytes to both systems. A fixture marked invalid composition is never committed by a phase and never enters the measured population: the commit-validation canary offers it once before and once after the measured window, to see whether the server refuses it as the declared posture says it should.

The pack's own description, as every record carries it

Reproduces the openEHR community's vital-signs benchmark harness (<https://discourse.openehr.org/t/17224>) and measures the same work a second way. The write phase creates 100 EHRs and commits the same Vital signs composition 1,000 times into each with Prefer: return=identifier, on one worker, and reports bulk-load throughput plus the whole-loop milliseconds-per-composition average the thread quotes, labelled closed-loop. The read phase then runs twice: read_walk is the sequential walk over every committed composition, seven GETs each (latest, version_at_time, the VERSIONED_COMPOSITION, its latest version, its version at that instant, one version by id, and the revision history), reporting the whole-loop microseconds-per-request average, labelled closed-loop; read_open_loop offers the same seven reads as an arrival schedule pinned at 200/s for 60s after a 15s warmup, which is where the coordinated-omission-free percentiles come from. The pinned rate is part of this pack version: changing it changes the work and bumps the version. Every version_at_time read addresses one instant captured after the write phase finished, which every seeded version predates, so it selects the same versions the harness's own start-of-run instant selects. Fixture provenance: the operational template is the vendored CKM export for template id 'Vital signs' (CKM cid 1013.26.380), byte-identical; the composition is the attachment on post 8 of that thread, byte-identical. Both are pinned by sha256 and verified at load. This pack version pins a failed-arrival ceiling of 0.01: a record in which any repetition, phase and operation loses a larger share of its arrivals, on the target or on any baseline, is not submittable, because percentiles taken over failed arrivals measure the failure rather than the system.

smoke version 1.0.0

seed 6216465318106710868 · 2 phases · failed-arrival ceiling 0.01 · drive it with --pack smoke

seed closed-loop, runs until it is finished mixed warmup 10s, discarded · measured 60s at 50 arrivals/s

The seed phase builds the population. It creates 200 EHRs through the public API and commits the same composition 5 times into each, on 8 workers, leaving 1,000 compositions behind. Every later phase reads and writes against exactly that population. The phase is closed-loop: the next request goes out once the previous one has answered, so it reports bulk-load throughput and never a latency claim.

The mixed phase offers arrivals on a fixed schedule: 50 a second for 60 seconds, after a 10-second warmup whose 500 arrivals are dispatched and then discarded. 3,000 arrivals are measured. They fire at their planned instants whether or not an earlier request has come back, and every latency is measured from the planned instant, so a server that stalls shows the stall in its percentiles instead of quietly receiving fewer requests.

What the mixed phase offers, and why each one is in the mix
OperationRequestShareOfferedWhat it probes
create_compositionPOST /ehr/{ehr_id}/composition2010/sthe commit path, measured while reads compete with it
get_composition_latestGET /ehr/{ehr_id}/composition/{uid}3015/sthe latest-version composition read
get_ehrGET /ehr/{ehr_id}2010/sthe EHR resource read, the cheapest addressed read the API offers
get_ehr_statusGET /ehr/{ehr_id}/ehr_status157.5/sthe status read, which reaches a second versioned object in the same EHR
adhoc_query_uidPOST /query/aql157.5/san EHR-scoped projection, so the query path is exercised beside the direct reads

A run against this pack declares one posture profile, which says what was switched on behind the numbers. minimal is what a run takes when it names none.

The posture profiles smoke defines, and what each one declares
ProfileWhat it switches onDeclares
minimal defaultThe bare spec-conformant surface: no audit trail, unsigned versions, commits validated against the operational template, uncompressed responses, one tenant.audit off
commit_validation template
compression off
tenancy single
version_signing none
The bytes it offers, and where they came from

blood_pressure.opt · operational template · 10.2 KiB · application/xml

Authored in this repository for the smoke pack: an ADL 1.4 operational template with template id 'cnf.blood_pressure', rooted at openEHR-EHR-COMPOSITION.minimal.v1 and constraining openEHR-EHR-OBSERVATION.blood_pressure.v2. It exists to give the smoke pack a small upload, and it is not derived from any published library.

sha256 97549fb2ab7ca36b9baa1cc86e857ef82924927a42140dfd3fd09a05dd83d006

bp_composition.json · composition · 3.4 KiB · application/json

Authored in this repository for the smoke pack: a canonical-JSON COMPOSITION declaring template id 'cnf.blood_pressure', carrying one POINT_EVENT with a systolic and a diastolic DV_QUANTITY in mm[Hg].

sha256 9eaea10c5171d1f4648c8e932a21ce624312a2cad98f49115f35efbbb344a3ce

bp_composition.missing_composer.json · invalid composition · 3.4 KiB · application/json

Derived in this repository from bp_composition.json by deleting the mandatory COMPOSITION.composer member and nothing else, so a server that validates a commit against the reference model refuses it.

sha256 602039bed3f3daf060152af6034baf6d7ce74fde6ec77e8ff1cc89eda2b3e0b3

Each digest is verified when the pack loads, so a run refuses to start if a single byte of any fixture has moved. Two records that name the same pack version therefore offered the same bytes to both systems. A fixture marked invalid composition is never committed by a phase and never enters the measured population: the commit-validation canary offers it once before and once after the measured window, to see whether the server refuses it as the declared posture says it should.

The pack's own description, as every record carries it

One blood-pressure template, a small EHR corpus, and a mixed open-loop phase over the read, write and query surface. This pack version pins a failed-arrival ceiling of 0.01: a record in which any repetition, phase and operation loses a larger share of its arrivals, on the target or on any baseline, is not submittable, because percentiles taken over failed arrivals measure the failure rather than the system.

The operation vocabulary

Every request a pack is allowed to offer

A pack may only offer an operation from this closed list. A token outside it is refused when the pack loads, so a typo can never quietly become a different measurement.

TokenRequest
adhoc_query_aggregatePOST /query/aql
adhoc_query_ehr_scanPOST /query/aql
adhoc_query_filteredPOST /query/aql
adhoc_query_ordered_pagePOST /query/aql
adhoc_query_point_lookupPOST /query/aql
adhoc_query_populationPOST /query/aql
adhoc_query_uidPOST /query/aql
create_compositionPOST /ehr/{ehr_id}/composition
get_composition_at_timeGET /ehr/{ehr_id}/composition/{uid}?version_at_time={at_time}
get_composition_latestGET /ehr/{ehr_id}/composition/{uid}
get_ehrGET /ehr/{ehr_id}
get_ehr_statusGET /ehr/{ehr_id}/ehr_status
get_versioned_compositionGET /ehr/{ehr_id}/versioned_composition/{uid}
get_versioned_composition_revision_historyGET /ehr/{ehr_id}/versioned_composition/{uid}/revision_history
get_versioned_composition_version_at_timeGET /ehr/{ehr_id}/versioned_composition/{uid}/version?version_at_time={at_time}
get_versioned_composition_version_by_idGET /ehr/{ehr_id}/versioned_composition/{uid}/version/{version_uid}
get_versioned_composition_version_latestGET /ehr/{ehr_id}/versioned_composition/{uid}/version

A path shown with {ehr_id}, {uid}, {version_uid} or {at_time} has that value substituted per arrival from the seeded draw. The tool builds every request it sends from exactly these templates.

Getting on the board

What a record has to carry before it can be ranked

A run against your own deployment is useful on its own. Ranking one against other people's records asks for more, and a record that misses any of it stays valid for local work while naming what it is missing.

  • repetitions: at least 3 repetitions, because one repetition measures a moment rather than a system
  • baseline: at least one same-machine baseline, because an absolute number without an anchor describes the machine as much as the system
  • error_share: every repetition, phase and operation, on the target and on every baseline, at or below the pack's failed-arrival ceiling, because percentiles taken over failed arrivals measure the failure rather than the system

Both are decided by the tool from the record itself, and CI checks them again before a maintainer reads the numbers. The full procedure, including the file naming and the append-only rule over merged records, is in the submission guide.

A record for the board comes out of one command, driving one of the pack versions on this page.

One command, one record
# The credential is read from the environment; it never rides argv.
$ export VEREDICTUM_BENCH_PASSWORD=…
$ veredictum bench --base-url https://cdr.example/openehr/v1 \
      --auth basic --user <user> \
      --pack community-vitals --repetitions 3 --with-baselines \
      --out ./bench --label "Your CDR 1.2.3"

To read this page as data instead, the manifest it was generated from is bench-packs.json, and veredictum bench-packs --out DIR writes the same document from any build.