validateHE validation report (abridged)

Example cohort model

This example shows how a validateHE report is structured. It is abridged: a real report has more sections, an environment manifest, full checklist tables and a function inventory. Costs are in invented cost units; no currency is implied.

1. Executive summary (non-technical)

Model: Example cohort model: a cohort state-transition (Markov) model in R with four health states (Stable, Progressed, Complicated, Dead), comparing Strategy A with Strategy B in monthly cycles.
Version reviewed: Not applicable. This is a synthetic example; no repository exists.
Requested modules: 1 Transparency, 3 Verification, 5 External validation

Overall assessment: Fit only after revisions

The model runs, and most of its basic consistency checks pass. Two do not: the time spent in one health state is counted twice when QALYs are added up for Strategy B, and costs stay discounted when discounting is switched off. Both change the cost-effectiveness results, so the model should be corrected and checked again before it informs a decision.

Fixing these two issues and re-running the checks would address the main concerns. The remaining points are minor.

The five validation methods at a glance

Validation method	What it asks	Assessment
1. Transparency	Is the model documented well enough to understand and re-run it?	Minor concerns. The run instructions leave open which script produces the base case.
2. Face validity	Does the model look plausible to a (simulated) council of experts?	Not requested
3. Verification	Does the code actually do what the model claims?	Major concerns. Two core consistency checks failed, and both change the results.
4. Cross-validation	Do other models of the same problem reach similar answers?	Not requested
5. External validation	Do the model's predictions match observed real-world data?	Not assessable. No observed data was supplied, so nothing was compared and no values were filled in from general knowledge.

Top findings in plain language

When we set every quality-of-life weight to 1, QALYs should equal life-years exactly. In Strategy B they were 4.1% higher, because the time spent in one health state is counted twice (finding BB-001).
When we set the discount rate to 0, discounted and undiscounted results should match. QALYs matched, but discounted costs were still 11.7% lower, because costs use a fixed discount rate instead of the model input (finding BB-002).

The expert council's view (AI-simulated)

Not requested. The face-validity council was not part of this run.

Checklists at a glance

CDA-AMC (HTA model-validation checklist): 37 of 57 items met; 2 failed; 8 deferred to modules not requested.
TECH-VER (verification checklist): 57 of 84 tests met; 4 failed.
AdViSHE (validation-assessment tool): 7 of 15 items met and 1 partly met; 3 not tested because no observed data was supplied.
CHAOS (open-source model checklist): 9 of 10 items met and 1 partly met.
CHEERS 2022 (reporting standard): not applicable; there is no manuscript in the documentation set.

2. Technical body

2.1 Scope, disclosures, and how to read this report

What was and wasn't assessed. Modules 1 (transparency), 3 (verification) and 5 (external validation) were requested. Modules 2 (face validity) and 4 (cross-validation) were not; the checklist items they own are counted as Deferred, not skipped.

Module 5 was requested without any observed data. Observed values are never filled in from general knowledge, so Module 5 is Not assessable (internal verdict: Inconclusive) and its checklist items are Not-tested.

AI-review disclosure. This review was carried out by AI reviewers. Each verification pass ran as an isolated reviewer, so a coding-style issue cannot bleed into a structural-error verdict. Read the findings as AI-assisted.

Integrity. No file in the reviewed repository was modified: all testing ran on a temporary copy, and a before/after content hash confirmed the original was unchanged. No fixes were applied. In this synthetic example no repository and no hash values exist.

Verdict vocabulary. The overall assessment uses a fitness-for-purpose scale: Fit for the stated decision problem, Fit with minor revisions, Fit only after revisions, Not yet fit, Not assessable on the reviewed evidence. Each method uses a concerns scale: No material concerns, Minor concerns, Major concerns, Critical concerns, Not assessable. Checklist items keep the instruments' own vocabulary: Pass, Partial, Fail, NA, Not-tested, Deferred.

Finding IDs. TR- findings come from the transparency reviewers (2.2), BB- from black-box testing and WB- from white-box inspection (2.3). Every finding with a recommended change appears in Appendix A.

"For clarification". Where a finding depends on context the review could not see, it is phrased as a question to the model's authors. An author response may resolve it without a change to the model.

2.2 Transparency results

TR-001: For clarification: which script produces the base case?

Severity: Minor
Verdict: Partial
Module 1, documentation completeness

Test: Run guidance review: the documentation should let a reader re-run the base case.
Inputs: Repository README and the supplied methods note.
Expected: The documentation names the script that produces the base case and the order in which the scripts run.
Observed: Two scripts each look like an entry point, and the README does not say which one produces the reported base case. Could the authors confirm?
Checklist items: CHAOS Run guidance: Partial (Basic)
Location: README.md
Recommended fix: State in the README which script produces the base case and in which order the scripts run. not applied

2.3 Verification results

Black-box tests change inputs and check the outputs against expectations set before the run.

BB-001: QALYs exceed life-years when every utility is set to 1

Severity: Major
Verdict: Fail
Module 3, black-box test BB-U1

Test: Utilities = 1: with every health-state utility set to 1, QALYs must equal life-years, discounted and undiscounted.
Inputs: All health-state utilities set to 1 for both strategies; all other inputs at base case.
Expected: QALYs equal life-years for Strategy A and for Strategy B.
Observed: Strategy A: 7.92 QALYs and 7.92 life-years. Strategy B: 8.71 QALYs and 8.37 life-years, so QALYs are 4.1% higher. The excess equals the time Strategy B spends in the Complicated state, which is added to the QALY total twice.
Checklist items: CDA-AMC 38; TECH-VER TV-3.2-07
Location: R/outcomes.R:48
Recommended fix: Count the QALYs accrued in the Complicated state once in the Strategy B total, then re-run the utilities = 1 test. not applied

BB-002: Costs are still discounted when the discount rate is 0

Severity: Major
Verdict: Fail
Module 3, black-box test BB-D0

Test: Discount rate = 0: with both discount rates set to 0, discounted results must equal undiscounted results.
Inputs: Cost and outcome discount rates set to 0.
Expected: Discounted costs and QALYs equal undiscounted costs and QALYs for both strategies.
Observed: QALYs match exactly. Discounted costs are 11.7% below undiscounted costs in both strategies, so the cost discounting step does not read the discount-rate input.
Checklist items: CDA-AMC 39; TECH-VER TV-3.3-14
Location: R/costs.R:23
Recommended fix: Take the cost discount rate from the model inputs instead of the fixed value in the cost discounting step, then re-run the discount rate = 0 test. not applied

BB-003: A cost remains when every cost input is set to 0

Severity: Minor
Verdict: Fail
Module 3, black-box test BB-C0

Test: Costs = 0: with every cost input set to 0, no costs may accumulate.
Inputs: All cost inputs set to 0 for both strategies.
Expected: Total costs of 0 for Strategy A and for Strategy B.
Observed: Strategy A: 0 cost units. Strategy B: 312 cost units per patient, from a one-off cost on entry to the Complicated state that is written into the code instead of read from the inputs.
Checklist items: TECH-VER TV-3.2-10
Location: R/costs.R:61
Recommended fix: Move the one-off Complicated-state entry cost into the model inputs, so it responds to cost changes and can be varied in sensitivity analysis. not applied

Other black-box tests in this run passed:

Test	ID	Verdict
The base case runs without error	BB-BASE	Pass
Mortality = 0: nobody dies and undiscounted life-years equal the time horizon	BB-M0	Pass
Swapping the two strategies' inputs swaps their results	BB-SW	Pass
Doubling the effect difference roughly doubles the incremental effects	BB-2X	Pass
Cohort conservation: state occupancy sums to 1 in every cycle	BB-TR	Pass
Absorbing state: occupancy of Dead never decreases	BB-TR	Pass
Determinism: running the base case twice gives identical results	BB-RE	Pass

White-box inspection reads the critical calculations line by line.

WB-001: Annual rate converted to a monthly probability by simple division

Severity: Minor
Verdict: Fail
Module 3, white-box inspection of time-unit conversion

Test: A rate r over a cycle of length t converts to the probability p = 1 − exp(−r·t).
Inputs: Annual rate of moving from Progressed to Complicated: 0.083. Cycle length: 1 month (t = 1/12 year).
Expected: p = 1 − exp(−0.083/12) = 0.006893 per cycle.
Observed: The code uses p = r·t = 0.006917 per cycle, 0.35% too high. The effect is small at this rate but grows with larger rates or longer cycles.
Checklist items: TECH-VER TV-3.2-21
Location: R/transitions.R:17
Recommended fix: Convert the annual rate with p = 1 − exp(−r·t). not applied

2.4 External-validation results

Assessment: Not assessable (internal verdict: Inconclusive).

Module 5 was requested, but no observed data was supplied. No predictions were compared, and no observed values were taken from general knowledge of the literature. The AdViSHE items on comparison with empirical data (D4, D4.A, D4.B) are Not-tested.

To make this module assessable, supply an aggregated table of observed values (endpoint, arm, time point, observed value, confidence interval, denominator) and state for each source whether the model was fitted to it.

2.5 Checklist rollup

Checklist	Pass	Partial	Fail	NA	Not-tested	Deferred
CDA-AMC Model Validation Tool (2024)	37	0	2	6	4	8
TECH-VER (2019)	57	0	4	9	8	6
AdViSHE (2016)	7	1	0	0	3	4
CHAOS v0.9 (draft, 2026)	9	1	0	0	0	0
CHEERS 2022 (conditional)	0	0	0	28	0	0

Deferred: owned by a module that was not requested. NA: the item cannot apply to this model. CHEERS 2022 is a manuscript reporting standard and counts as NA here because the documentation set has no manuscript. AdViSHE C1 is Partial because the only external review of the implemented model is this AI-assisted review.

Appendix A: Recommended changes and points for clarification (not applied)

validateHE modified nothing. The changes below are recommendations for the model's maintainers, in descending order of severity. Rows that open with "For clarification" are questions an author response may resolve without a change to the model.

ID	File:line	Severity	Issue	Recommended change	Checklist items
BB-001	R/outcomes.R:48	Major	QALYs exceed life-years when every utility is set to 1	Count the QALYs accrued in the Complicated state once in the Strategy B total, then re-run the utilities = 1 test. not applied	CDA-AMC 38; TECH-VER TV-3.2-07
BB-002	R/costs.R:23	Major	Costs are still discounted when the discount rate is 0	Take the cost discount rate from the model inputs instead of the fixed value in the cost discounting step, then re-run the discount rate = 0 test. not applied	CDA-AMC 39; TECH-VER TV-3.3-14
BB-003	R/costs.R:61	Minor	A cost remains when every cost input is set to 0	Move the one-off Complicated-state entry cost into the model inputs, so it responds to cost changes and can be varied in sensitivity analysis. not applied	TECH-VER TV-3.2-10
WB-001	R/transitions.R:17	Minor	Annual rate converted to a monthly probability by simple division	Convert the annual rate with p = 1 − exp(−r·t). not applied	TECH-VER TV-3.2-21
TR-001	README.md	Minor	For clarification: which script produces the base case?	State in the README which script produces the base case and in which order the scripts run. not applied	CHAOS Run guidance: Partial (Basic)

Request a spot in the free beta