ARKA ML model card
Glass-box EBM concordance calibrator, training data, metrics, limitations, and a voluntary HTI-1 DSI source-attribute sheet.
Last updated: August 22, 2026
Model Card — ARKA Guideline-Concordance EBM
Model card format follows Model Cards for Model Reporting (Mitchell et al., 2019). This document accompanies the ARKA CDS Hooks ML service (ml-service/) and the Feature Rationale Catalogue (lib/cds-platform/ml/feature-catalog.ts).
Model details
| Field | Value |
|---|---|
| Model name | ARKA Guideline-Concordance Calibrator |
| Version | 2.0.0-ebm |
| Type | InterpretML Explainable Boosting Machine (ExplainableBoostingClassifier) |
| Output | Calibrated guideline-concordance probability ∈ [0, 1] only |
| Not an output | Appropriateness score (1–9) — that comes from the transparent AIIE 2.0 core (lib/aiie-v2) |
| Feature vector | aiie_raw_posterior (transparent-core posterior) + 23 structured case features |
| Explainability | Exact per-feature shape functions exported to model/ebm_shape_functions.json (Criterion 4) |
| Maintainer | ARKA Health — arkahealth.com |
Retired: model/trained_model.json (XGBoost appropriateness regressor) is deprecated. It was a self-referential oracle trained to reproduce hand rules. It is not loaded for scoring. The rule-based fallback is the faithful reference when the EBM is unavailable.
Intended use
- Primary use: Map the AIIE 2.0 raw posterior plus structured case features to a calibrated concordance (or, later, denial) probability for operational thresholds and glass-box review.
- Users: Licensed clinicians and clinical staff; the probability is advisory context alongside the transparent-core appropriateness rating and guideline citations.
- Integration: Consumed by ARKA CDS Hooks services. The appropriateness number shown to users is produced by
lib/aiie-v2(or the documented rule-based fallback), not by this EBM.
Out-of-scope use (do not use for)
- Appropriateness oracle — do not treat EBM output as a 1–9 appropriateness score.
- Clinical outcome prediction presented as validated performance from synthetic self-consistency metrics.
- Time-critical alerts (stroke code, trauma activation, sepsis) without human review.
- Image interpretation or CAD on pixel data (FDA Criterion 1: no signal processing).
- Autonomous ordering without clinician review.
Training data
| Attribute | Description |
|---|---|
| Source (current) | Guideline-concordance labels derived from signed-off / ACR-aligned stewardship heuristics (bootstrap). Synthetic feature vectors smoke-test plumbing only. |
| Source (planned / AIIE 3.0) | Federated clinician overrides and downstream outcomes (PHI-free hashes) under change-control — see docs/AIIE_CHANGE_CONTROL.md. Until then, pretest/LR/threshold calibration remains concordance-anchored. |
| Sample size | ~5,000 bootstrap examples per training run (80% / 20% split) |
| Labels | Binary guideline concordance (not appropriateness score; not adjudicated outcomes) |
| PHI | None in bootstrap; future outcome data uses hashed identifiers only |
| Real-world data | Not yet used for weights; will refit only this calibration layer when available (change control) |
Explicit non-claim: Synthetic / self-consistency / concordance figures are never clinical validity. Do not cite bootstrap concordance metrics as clinical-performance evidence. Concordance ≠ outcomes until real outcomes exist.
Evaluation data and metrics
| Dataset | Role |
|---|---|
| Bootstrap held-out set | Smoke-test of EBM plumbing (model/evaluation/metrics.json) — guideline concordance only |
| Guideline-concordance calibration cohort | Built by model/calibrate.py from the human-signed-off knowledge matrix + signed-off ingest seed + CDR literature anchors (model/evaluation/calibration.json) |
EBM metrics (bootstrap held-out — NOT clinical validity):
| Metric | Description |
|---|---|
| AUC / Brier / log loss | Discrimination and calibration of P(concordance) |
| Accuracy @ 0.5 | Thresholded concordance label agreement (plumbing check) |
Every metrics artifact includes a caveat field stating these figures are not clinical validity.
AIIE 2.0 conformal calibration (guideline concordance — not outcomes)
Run: ml-service/.venv/bin/python ml-service/model/calibrate.py
Caveat: These figures measure guideline concordance (agreement with the human-signed-off matrix / signed-off ingest / CDR literature references) and interval coverage. They are not clinical outcome validity, denial-prediction accuracy, or patient-outcome claims.
| Metric | Value (held-out, representative run) | Notes | ||
|---|---|---|---|---|
| Concordance (\ | ŷ − ref\ | ≤ 1.5) | 0.875 | Guideline concordance rate |
| 90% credible-interval coverage | 0.865 | Bayesian posterior CI from scoreOrderV2 (in target [0.86, 0.94]) | ||
| Split-conformal set coverage (ref ∈ [ŷ ± q]) | 0.914 | Distribution-free (1−α) metric | ||
| ECE | 0.136 | Expected calibration error on confidence vs concordance | ||
| Calibration slope | 0.880 | OLS slope of reference ~ predicted (ideal = 1.0) |
Artifacts: model/evaluation/calibration.json, model/evaluation/conformal_table.json, model/evaluation/reliability_diagram.png. Runtime table: lib/aiie-v2/data/conformal_table.json.
AIIE 3.0 conformal risk control (important-miss rate — not outcomes)
Run: python ml-service/model/crc_calibrate.py [--cohort path] [--alpha 0.1]
Loss definition (explicit — no hidden objective):
| Symbol | Meaning | ||
|---|---|---|---|
s(x) | Retention-risk score ∈ [0, 1] (retentionRiskScore: CI width + band-boundary proximity) | ||
M | Important-miss label (reviewer flag, or concordance proxy: band disagree AND \ | ŷ−ref\ | > 2) |
λ | Abstain threshold: retain if s ≤ λ, defer if s > λ | ||
L_i(λ) | M_i · 1{s_i ≤ λ} — loss only when a retained case is an important miss (bounded by B=1, monotone in λ) |
CRC chooses λ̂ = max{λ : (n/(n+1)) R̂_n(λ) + B/(n+1) ≤ α} so that under exchangeability E[L_{n+1}(λ̂)] ≤ α (Angelopoulos, Bates, Fisch, Lei, Schuster, ICLR 2024).
Caveat: Bootstrap / concordance-proxy figures are not clinical outcome validity. Final abstention is v2.abstain OR crc.abstain (never less cautious than AIIE 2.0 interval-coverage abstain).
| Artifact | Path |
|---|---|
| Runtime table | lib/aiie-v3/risk_control_table.json |
| Evaluation copy | ml-service/model/evaluation/risk_control_table.json |
| TS API | lib/aiie-v3/risk-control.ts |
AIIE 3.0 decision-impact validation (P12 — not outcomes)
Run: python ml-service/model/validate_v3.py [--cohort path]
Training / calibration data: guideline-concordance labels from the human-signed-off knowledge matrix + signed-off ingest + CDR literature anchors; later federated clinician overrides and real-world outcomes under change-control. Until outcomes exist, every number below is concordance / decision-impact plumbing — not clinical validity.
What is measured (honest labels — §9 plan):
| # | Metric | Meaning |
|---|---|---|
| 1 | Decision-impact | Fraction of holdout cases where A3 lands in a different 1–9 band than A2, with direction (up/down) and reason buckets (e.g. low_p_mgmt_plus_radiation_harm, high_pretest_decisive_posttest) |
| 2 | Net benefit | Vickers decision-curve net benefit of the A3 imaging recommendation vs image-all / image-none defaults (concordance-proxy positives: reference ≥ 7) |
| 3 | Risk-control coverage | Empirical CRC important-miss risk under fitted λ — must be ≤ α (+ finite-sample tolerance) |
| 4 | Interval coverage & calibration | Carried from AIIE 2.0 calibration.json (90% CI / conformal set, ECE, reliability diagram, calibration slope) |
| 5 | Pretest / post-test calibration | ECE of glass-box p0 / p1 vs concordance-proxy labels, overall and per target condition |
| 6 | Selective-prediction | Risk–coverage curve: deferred cases are higher retention-risk; retained reasoned-impact / concordance reported as coverage drops |
| 7 | Subgroup equity | Concordance + net benefit + decision-impact by age band, sex, modality, region (multi-calibration closes gaps >5 pts at runtime — P11) |
| 8 | Ablations | v2-only → +pretest → +post-test → +decision → +harm → full 3.0; v2_only impact = 0; graceful-degrade A3 == A2 when uninformative |
| 9 | Prospective / outcomes | Deferred until federated overrides/outcomes exist — management-change concordance, denial-overturn lift, AUC vs approve/deny are not claimed |
| — | Degrade-to-v2 rate | Fraction of cases where the decision layer is uninformative and A3 mirrors A2 |
| — | Latency | Core = table arithmetic (µs); only network cost is cached/bounded P10 external context; CDS p95 budget 800 ms |
Artifact: model/evaluation/v3_validation.json. Promote gates (POST /api/ins/aiie/promote with engine: "aiie-3" / v3Validation) reject challengers that regress risk coverage, net benefit, subgroup concordance (>5 pts), degrade-to-v2 rate, or shadow divergence (409 + failingCheck). Health: GET /api/ins/aiie/health returns v3.championVersion, v3.decisionImpact, v3.riskCoverage, and v3.degradeRate in one call.
Pending real-world outcomes: management-change concordance, denial-overturn lift, and AUC vs approve/deny are not claimed until federated outcomes accrue. Concordance ≠ clinical validity.
Fairness and subgroup analysis
Subgroup performance is reported in calibration.json for the guideline-concordance holdout:
| Subgroup | Stratification |
|---|---|
| Age | Pediatric (<18), adult (18–64), older adult (65+) |
| Sex | Male / female |
| Modality | CT, MRI, radiograph, ultrasound, … |
| Region | Knowledge-matrix body region |
Target: Flag any subgroup >5 absolute percentage points off overall concordance. Flagged rows are listed under flagged_subgroups in calibration.json (investigate before treating concordance gaps as outcome disparities).
Known limitations
- Bootstrap labels are guideline-concordance, not adjudicated real-world outcomes.
- 23-feature vector plus posterior may omit site-specific pathways or free-text nuance.
- English-language, US guideline framing; not validated for non-US practice.
- Fallback mode: If
ebm_model.joblibis absent, the service uses the rule-based scorer as the faithful reference (more faithful than the retired XGBoost oracle) with lower stated confidence. - Shape functions are main-effect additive terms; interaction terms are disabled by default for independent reviewability.
- Clinical sign-off for catalogue rationales is tracked in
docs/CLINICAL_SIGN_OFF_LOG.md.
Regulatory posture
ARKA Imaging Intelligence Engine is positioned as an FDA Non-Device Clinical Decision Support tool under FD&C Act §520(o)(1)(E) (21st Century Cures Act). This model:
- Does not process medical images.
- Surfaces exact shape-function contributions and peer-reviewed / guideline-linked rationales (Criterion 4).
- Requires clinician responsibility for the final order decision.
Ethics and safety
- No autonomous patient-facing decisions.
- Predictions are advisory; overrides must remain available in the EHR workflow.
- Report safety or appropriateness concerns through the contact below before relying on the model in production.
Contact and issue reporting
- Issues / safety reports: https://arkahealth.com (contact form / support channel)
- Repository path:
ml-service/MODEL_CARD.md - Change control (PCCP-style):
docs/AIIE_CHANGE_CONTROL.md— what may auto-recalibrate vs clinician sign-off, shadow-before-promote, rollback - Catalogue source of truth:
lib/cds-platform/ml/feature-catalog.ts→npm run export:feature-catalog - Train:
python ml-service/model/train_ebm.py - Recalibrate (v2 conformal / EBM):
python ml-service/model/calibrate.py [--feedback-cohort path] [--refit-ebm] - Recalibrate (v3 CRC abstain):
python ml-service/model/crc_calibrate.py [--cohort path] [--alpha 0.1] - Validate (v3 decision-impact / NB / risk-coverage):
python ml-service/model/validate_v3.py [--cohort path]
Last updated: 2026-07-20. Regenerate EBM artifacts after catalogue or feature-engineering change. Re-run model/calibrate.py after matrix or CDR pack changes. Re-run model/crc_calibrate.py when important-miss labels accrue. Re-run model/validate_v3.py before any AIIE 3.0 promote. Feedback-driven refits follow docs/AIIE_CHANGE_CONTROL.md.
HTI-1 predictive DSI source attributes
ARKA is voluntarily answering a published question set — the ONC HTI-1 predictive decision-support source attributes under 45 CFR 170.315(b)(11) — even though ARKA is not a certified Health IT Module and is not obliged to publish this sheet.
This page does not claim or imply certification, conformance, or endorsement by ONC, the Joint Commission, or CHAI.
The deterministic knowledge matrix and coverage runtime are evidence-based decision support. The pretest/post-test probability model in lib/aiie-v3/ is predictive. Each attribute below names which component(s) it applies to.
Criterion 45 CFR 170.315(b)(11) · v1.0.0 · 22 August 2026 · 27 answered · 4 unanswered · machine artefact docs/dsi-source-attributes.json
This section is a set of links to existing artefacts, not a second set of claims. Attributes that cannot yet be answered honestly are marked unanswered with a reason — silence fails CHECK-DSI-1.
Category A
| # | Attribute | Answer |
|---|---|---|
| 1 | Intervention name Applies to: Both evidence-based matrix and predictive pretest/post-test model | ARKA Imaging Appropriateness CDS (evidence-based matrix + predictive pretest/post-test where enabled) Source: /docs/model-card · recorded 2026-08-22 |
| 2 | Intervention version Applies to: Both evidence-based matrix and predictive pretest/post-test model | Published matrix and engine versions on the model card and certainty coverage artefact; deployment-specific release noted at enablement. Source: |
| 3 | Developer name and contact Applies to: Platform / CDS delivery (cards, disclosure, governance) | ARKA Health, Inc. · security@getarka.health Source: /company · recorded 2026-08-22 |
| 4 | Last update date Applies to: Both evidence-based matrix and predictive pretest/post-test model | Source-attribute sheet dated 22 August 2026; matrix updates are release-noted and panel-recorded. Source: /governance/evidence-panel · recorded 2026-08-22 |
Category B
| # | Attribute | Answer |
|---|---|---|
| 5 | Intended use and purpose Applies to: Both evidence-based matrix and predictive pretest/post-test model | Support licensed clinicians selecting clinically appropriate diagnostic imaging from structured FHIR at order entry — Non-Device CDS under FD&C Act §520(o)(1)(E). Source: /docs/model-limitations · recorded 2026-08-22 |
| 6 | Cautioned out-of-scope use Applies to: Both evidence-based matrix and predictive pretest/post-test model | Does not analyse images or signals; not for time-critical triage; not for autonomous order placement or cancellation. Source: /docs/model-limitations · recorded 2026-08-22 |
| 7 | Target patient population Applies to: Both evidence-based matrix and predictive pretest/post-test model | Adults with orders for advanced diagnostic imaging where structured FHIR order-select context is available. Source: /docs/model-limitations · recorded 2026-08-22 |
| 8 | Target care setting Applies to: Platform / CDS delivery (cards, disclosure, governance) | Inpatient and outpatient order entry via CDS Hooks in certified EHR workflows. Source: /docs/integrations · recorded 2026-08-22 |
Category C
| # | Attribute | Answer |
|---|---|---|
| 9 | Output type Applies to: Both evidence-based matrix and predictive pretest/post-test model | CDS suggestion cards with appropriateness support, optional pretest/post-test probabilities, citations, and FDA Non-Device CDS disclosure. Source: /docs/model-card · recorded 2026-08-22 |
| 10 | How output is generated Applies to: Both evidence-based matrix and predictive pretest/post-test model | Evidence-based: deterministic knowledge-matrix lookup and coverage runtime. Predictive: glass-box pretest/post-test probability model in lib/aiie-v3/. Neither path is an autonomous order. Source: |
| 11 | Basis for recommendation Applies to: Evidence-based — deterministic knowledge matrix and coverage runtime | Named guideline and literature anchors on the Source Register; GRADE certainty on the certainty census; feature attributions on demand. Source: /sources · recorded 2026-08-22 |
| 12 | Human-readable output label Applies to: Platform / CDS delivery (cards, disclosure, governance) | CDS card suggestion labels per the published card language style guide (supportive, non-coercive). Source: |
Category D
| # | Attribute | Answer |
|---|---|---|
| 13 | Type of data used to develop Applies to: Both evidence-based matrix and predictive pretest/post-test model | Published guidelines and literature for the knowledge matrix; synthetic and de-identified cohorts for calibrator development — no PHI in training. Source: /sources · recorded 2026-08-22 |
| 14 | Description of training data Applies to: Predictive — pretest/post-test probability model (lib/aiie-v3/) | Matrix ratings are evidence-anchored, not fitted on PHI. ML calibrator training data are described on the model card (synthetic cohort). Source: /docs/model-card · recorded 2026-08-22 |
| 15 | Data provenance and collection Applies to: Evidence-based — deterministic knowledge matrix and coverage runtime | Source Register entries carry citation, retrieval date, and COI class; panel records seal certainty changes. Source: /sources · recorded 2026-08-22 |
| 16 | Demographic representation in training Applies to: Predictive — pretest/post-test probability model (lib/aiie-v3/) |
Related: /docs/model-limitations · recorded 2026-08-22 |
| 17 | Known data gaps or limitations Applies to: Both evidence-based matrix and predictive pretest/post-test model | Published on the model-limitations page: synthetic validation today; very_low GRADE share disclosed on the certainty burn-down; site outcome pairing not yet evaluated. Source: /docs/model-limitations · recorded 2026-08-22 |
| 18 | Process for updating training data Applies to: Evidence-based — deterministic knowledge matrix and coverage runtime | Standing Evidence Panel reviews certainty and anchors; matrix emits are versioned; corrections are append-only code changes. Source: /governance/evidence-panel · recorded 2026-08-22 |
Category E
| # | Attribute | Answer |
|---|---|---|
| 19 | Maintenance schedule Applies to: Both evidence-based matrix and predictive pretest/post-test model | Panel session schedule and review queue are published; rule-library and model updates are release-noted with no silent changes. Source: /governance/evidence-panel · recorded 2026-08-22 |
| 20 | Process for retraining or recalibration Applies to: Predictive — pretest/post-test probability model (lib/aiie-v3/) | ARKA does not auto-retrain from local drift. Recalibration requires an explicit change-control release; drift breaches alert and notify — they do not retrain. Source: /monitoring · recorded 2026-08-22 |
| 21 | How often validity and fairness are monitored in local data Applies to: Both evidence-based matrix and predictive pretest/post-test model | Method published on /monitoring (certainty burn-down, calibration status, drift under three strategies). Local paired outcomes are not yet evaluated — the calibration block states the data ask explicitly. Source: /monitoring · recorded 2026-08-22 |
| 22 | Override and feedback capture Applies to: Platform / CDS delivery (cards, disclosure, governance) | CDS Hooks feedback endpoint and Module 3 accountable-justification receipts (hash only — no justification text stored by ARKA). Source: /api/cds-services/feedback · recorded 2026-08-22 |
| 23 | Incident and safety-event intake Applies to: Platform / CDS delivery (cards, disclosure, governance) | AI governance packet Element 5 documents the safety-event path; never-do catalog names hard refusals. Source: /governance/ai · recorded 2026-08-22 |
Category F
| # | Attribute | Answer |
|---|---|---|
| 24 | External validation and who performed it Applies to: Both evidence-based matrix and predictive pretest/post-test model |
Related: /docs/model-limitations · recorded 2026-08-22 |
| 25 | Quantitative performance measures for validity in test data Applies to: Both evidence-based matrix and predictive pretest/post-test model | ~87.5% guideline concordance on a held-out signed-off scenario cohort — agreement with the knowledge matrix, not a clinical-endpoint claim. Calibration against local outcomes: not yet evaluated (/monitoring). Source: /docs/model-limitations · recorded 2026-08-22 |
| 26 | Quantitative performance measures for fairness in test and external data Applies to: Predictive — pretest/post-test probability model (lib/aiie-v3/) |
Related: /docs/model-limitations · recorded 2026-08-22 |
| 27 | Known failure modes Applies to: Both evidence-based matrix and predictive pretest/post-test model | Published on /docs/model-limitations (missing context, out-of-scope orders, very_low certainty, feedback-loop contamination of unweighted drift). Source: /docs/model-limitations · recorded 2026-08-22 |
| 28 | Mitigations and harm containment Applies to: Platform / CDS delivery (cards, disclosure, governance) | Non-blocking CDS; no order authority; FDA disclosure on every card; never-do catalog; small-cell suppression; no auto-retrain from drift. Source: /governance/never-do · recorded 2026-08-22 |
Category G
| # | Attribute | Answer |
|---|---|---|
| 29 | Intervention risk management (validity, reliability, robustness, fairness, intelligibility, safety, security, privacy) Applies to: Platform / CDS delivery (cards, disclosure, governance) | Crosswalked on /governance/ai (RUAIH / NIST AI RMF). Fairness remains unanswered as a measured attribute (see #16 and #26). Security and privacy: /security and data-flow statement. Source: /governance/ai · recorded 2026-08-22 |
| 30 | Clinician transparency statement Applies to: Platform / CDS delivery (cards, disclosure, governance) | FDA Non-Device CDS disclosure on every card detail field — lib/compliance/fda-disclosure.ts. Source: |
| 31 | FAVES-aligned fairness summary Applies to: Predictive — pretest/post-test probability model (lib/aiie-v3/) |
Recorded 2026-08-22 |