validateHE: validation of R-based health-economic models
Free while in beta. Expanded model testing starts November 2026; we schedule models in the order requests arrive.
Errors in health-economic models are common, and validation is rarely reported.
Expert review is scarce and slow. AI agents can read, run and test model code, yet a PubMed search in October 2026 found no peer-reviewed study of AI for validating health-economic models; studies of language models so far build or replicate models. [5, 6]
Human and AI validation might complement each other. People bring scoping, subject-matter knowledge, judgement and experience; agents may be stronger on code and detail.
A model can pass verification and still be wrong about the world.
That is why validateHE covers all five validation types, not only the code.
validateHE is an AI agent skill: instructions, checklists and base-R scripts that an AI coding agent follows. It runs in Claude Code and OpenAI Codex, and other agentic tools that support skills may be able to run it too.
It validates R-based health-economic models (cohort state-transition (Markov), decision tree, discrete event simulation, partitioned survival and individual-level microsimulation) and follows the five validation types of the ISPOR-SMDM Modeling Good Research Practices Task Force. [1]
Is the model documented clearly enough to review?
Documentation reviewers check the README, dossier and code comments for completeness and consistency, and score CHEERS 2022 when a manuscript is supplied.
You bring: Your code and documentation; a manuscript or preprint adds CHEERS 2022.
Tested on: production modelsDo the structure, inputs and outcomes look plausible?
A simulated evidence review group: six AI reviewers with separate roles each write an independent critique, blind to the others; an optional deliberation round follows.
You bring: Nothing beyond your code and documentation.
Tested on: production modelsDoes the code do what the modelers intended, correctly?
Black-box tests (including unit tests of each function), white-box review with root-cause tracing, and a coding-standards review.
You bring: Nothing beyond your code and documentation.
Tested on: production models (partitioned survival, decision tree); tutorial models only (Markov, DES, microsimulation)Do other models of the same decision problem reach compatible answers?
Compares results with a supplied comparator model or published results: corroboration, not proof.
You bring: A comparator model of the same decision problem, or its published results.
Tested on: public tutorial models onlyDoes the model reproduce what actually happened?
Compares predictions with supplied observed data, such as life tables, registries or trial data.
You bring: Observed data, as an aggregated table (no patient-level data).
Tested on: public tutorial models onlyFindings are mapped item by item to four published checklists (CDA-AMC, TECH-VER, AdViSHE and CHAOS), plus CHEERS 2022 when the documentation includes a manuscript. Every verdict cites the test it rests on. [8, 9, 10, 11, 12]
Where it has been tested: transparency, face validity and the verification of partitioned survival and decision-tree models have been used on production models (the first results below). The verification of Markov, DES and microsimulation models, cross-validation and external validation have so far been tested only on public tutorial models.
| Checklist | M1 transparency | M2 face validity | M3 verification | M4 cross-validation | M5 external validation |
|---|---|---|---|---|---|
| CDA-AMC Model Validation ToolHTA model-validation checklist, 53 items | Scored by M1 transparency | Scored by M2 face validity | Scored by M3 verification | Scored by M4 cross-validation | Scored by M5 external validation |
| TECH-VERVerification checklist in five stages | Not scored by M1 transparency | Scored by M2 face validity | Scored by M3 verification | Scored by M4 cross-validation | Scored by M5 external validation |
| AdViSHEValidation-assessment tool, parts A to E | Scored by M1 transparency | Scored by M2 face validity | Scored by M3 verification | Scored by M4 cross-validation | Scored by M5 external validation |
| CHAOS (draft)Checklist for open-source models | Scored by M1 transparency | Not scored by M2 face validity | Scored by M3 verification | Not scored by M4 cross-validation | Not scored by M5 external validation |
| CHEERS 2022Reporting standard, 28 items | Scored by M1 transparency only when a manuscript is supplied | Not scored by M2 face validity | Not scored by M3 verification | Not scored by M4 cross-validation | Not scored by M5 external validation |
Scores items. Only when a manuscript or preprint is supplied. M1 transparency, M2 face validity, M3 verification, M4 cross-validation, M5 external validation. Items owned by a module that was not run are reported as Deferred; Excel-specific CDA-AMC items are scored through their R equivalents.
Module 2 simulates an HTA evidence review group: six AI reviewers with deliberately separate roles judge whether the model's structure, inputs and outcomes are plausible.
Each member reviews an evidence packet built from the documentation, the code inventory, inputs and outputs, and writes a structured critique, blind to the others: what fits its practice, what raises concern, and what it would need before signing off.
The members then read each other's critiques, respond, and converge or record dissent. Whether this round adds value beyond the independent critiques is still being tested. The council runs last, so it also reads the verification, cross-validation and external-validation findings.
The consensus view, dissent named by role and open questions for the authors go into the report, with recommended fixes that are not applied.
Disease pathway, clinical events, mortality, health outcomes and treatment pathway.
Model type, state and event structure, cycle length, extrapolation and structural assumptions; reads the technical findings.
Whether the economics is correct: perspective, discounting, costs, QALYs and incremental analysis.
Whether an appraisal body would accept it: reference case, comparators, evidence standards and uncertainty, for a named jurisdiction.
Patient-important outcomes, adverse events, treatment burden and quality of life.
Affordability, budget impact, resource use and implementation.
The council is AI-simulated, not a panel of human experts, and the report says so. Checklist items that ask whether human experts were consulted stay capped unless your materials document a real human expert review.
The first use on four "production" R models (three partitioned survival models and one decision tree) returned 287 actionable findings, 20 of them critical.
| Module | Critical | Major | Minor |
|---|---|---|---|
| M1 Transparency | |||
| M2 Face validity | |||
| M3 Verification | |||
| Total |
Share a current project to find bugs before reviewers do, or an older model with its documentation just to try validateHE.
We share the skill with all its materials (e.g., checklists) via GitHub or e-mail.
Running it yourself needs R, Quarto and an AI coding agent such as Claude Code or OpenAI Codex.
You get a full report with evidence-cited findings and recommended fixes; in return, a short evaluation survey.
R models now; Excel and other languages soon.
How a validateHE report is laid out: a short non-technical executive summary, a technical body with the evidence behind each finding, and an appendix of recommended changes that were not applied.
The example is fully invented. It describes no real model and no real result.
ISPOR Europe 2026 handout, accompanying poster EE146
Methods and checklist coverage, first results on four "production" R models, limitations, a synthetic report excerpt, FAQ and references. Prepared for the conference in the University of Oslo poster format.
We are still developing our processes for the full range of projects, from open-source models of academic projects to confidential and sensitive models of commercial organizations. During the beta, it works like this.
Questions about any of this are welcome: .
Now
R models, five validation modules, Claude Code and OpenAI Codex.
From November 2026
All five modules, with cross- and external validation where comparator or observed data exist. Prospective testing with new model teams, reports rated by survey; retrospective testing on published open-source models.
Soon
Beyond R, for example Excel and Python.
Model owners confirm or reject each finding and its grading. Independent human reviewers review a subset of the same models, so their findings can be compared with the agents'. Models with deliberately planted errors show what the agents catch and miss. The council is compared with and without its deliberation round.
Dates are targets.
validateHE is built by Benjamin P. Geisler, M.D., M.P.H., at the Department of Health Management and Health Economics, University of Oslo.
It was presented at ISPOR Europe 2026 as poster EE146; the handout describes the methods and first results. Questions and ideas are welcome: .
The model is run and its outputs are tested without reading the code, against expectations set before the run. Examples: with every utility set to 1, QALYs must equal life-years; with all costs set to 0, no costs may accumulate; with the discount rate at 0, discounted and undiscounted results must match; swapping two strategies' inputs must swap their results; the cohort must be conserved in every cycle; and the base case must give identical results when run twice. It also writes and runs unit tests for each function on the path from inputs to results.
The critical calculations are read line by line: transition-matrix construction, discount timing and rate, time-unit conversions, survival-parameter extraction, cost and utility structuring, and half-cycle correction. Results are traced back to their inputs and inputs forward to results, and the root cause of each black-box failure is traced through a map of which function calls which.
Consistency first, and patterns that make a model hard to audit or hide its failures: hidden global variables, duplicated definitions that drift apart, and silently swallowed errors are graded above style issues. It also checks documentation, modularity and reproducibility: whether a lockfile exists and whether the base case runs and replicates. Style is judged against a configurable profile: by default the tidyverse style guide with DARTH naming, or your team's own conventions.
Claude Code and OpenAI Codex. validateHE is packaged as a standard skill folder, so other agentic coding tools that support skills may be able to run it as well; those are untested.
No. It works on a temporary copy, checks by before/after content hash that the original is unchanged, and never applies fixes; they are listed in the report as recommendations.
Models written in R: cohort Markov, decision tree, partitioned survival, discrete event simulation and microsimulation. Excel and other languages are planned.
No. It is AI-simulated, and the report says so. Checklist items that ask for human expert review stay capped unless your materials document one.
No. It is a review aid: every finding cites its test and file location so a person can check it, and the report states what it could not test in that run, and why.
Then cross-validation and external validation are reported as not assessable. Values are never filled in from general knowledge.
The CDA-AMC Model Validation Tool, TECH-VER, AdViSHE and CHAOS, plus CHEERS 2022 when the documentation includes a manuscript or preprint. Items owned by a module that was not run are reported as Deferred. The checklists are listed under Sources.
A report in HTML and PDF: a plain-language executive summary, a technical body with the evidence behind each finding, and appendices with the recommended fixes, a function inventory and dependency map, and the council minutes.
Nothing during the beta. In return, we ask for a short evaluation survey.
A full run takes a couple of hours of agent time. If we run it for you, we agree a turnaround with you.
The beta is an academic evaluation at the University of Oslo, and its results will be published. What follows the beta is not yet decided.
Participants are acknowledged in the evaluation paper. Those who contribute substantively, for example by confirming and grading findings, can be co-authors under the ICMJE criteria. Your model becomes a case study only with your written permission.
Yes. There is a fully synthetic example report, and the ISPOR Europe 2026 handout shows methods and first results.
Tell us a little about yourself and your model. Prefer e-mail? Write to .
