Model limitations & oversight

The number we lead with

ARKA publishes ~87.5% guideline concordance on a held-out, human-signed-off scenario cohort from AIIE 2.0 conformal calibration. Concordance means agreement with the signed-off knowledge matrix / ingest / CDR literature references (absolute residual ≤ 1.5 on the 1–9 scale) — not a sensitivity/specificity claim for any clinical endpoint, and not the retired self-referential oracle metrics.

Synthetic self-consistency labels (validation plumbing) are used only to smoke-test feature extraction and dashboards. They are never equated with clinical validity.

We validate concordance first by design: model development and internal benchmarking use no PHI. De-identification and limited-data-set handling follow ARKA-PRIV-002 (see the data-flow statement). Real-world labeled outcomes are addressed in Tier 2 and Tier 3 of the evidence ladder.

Measured sandbox metrics: validation dashboard. Regulatory context: Trust center.

Known failure modes

Honest limits — aligned with validation dashboard disclaimers and the themes in our gated model card, without exposing proprietary training details:

• Sparse or atypical structured inputs — rare presentations under-represented in published guidelines may not trigger a rule or may receive weak model signal.
• Documentation quality dependence — garbage-in: incomplete FHIR prefetch or missing problem-list context reduces appropriateness signal quality.
• Payer-policy drift — LCD/NCD and plan-specific medical-necessity rules can change between rule-library updates; ARKA-INS administrative checks lag until the library is refreshed.
• Synthetic-to-real distribution shift — the primary reason Tier 2 (retrospective real-world data) and Tier 3 (prospective pilot) exist on the evidence ladder.
• Optional concordance calibrator — a glass-box EBM maps the AIIE 2.0 posterior plus case features to a calibrated concordance probability (exact shape functions). When unavailable, ARKA returns rules-only output — the faithful reference floor, not a crash. Appropriateness remains guideline-anchored from the transparent core regardless of calibrator status.

Design mitigations

• Non-blocking / no order authority — ARKA cannot place, cancel, modify, or hold orders. If ARKA is unreachable, the EHR proceeds as before (fail-open).
• Silent when no rule fires — zero added workflow when no guideline-anchored rule applies; alert-fatigue guardrails are a pilot KPI on /outcomes.
• SHAP reasoning + cited guideline on every card — supports FDA Non-Device CDS Criterion 4 independent review before the clinician acts.
• No time-critical or triage use — intended use explicitly excludes time-critical alerts and triage. This aligns with FDA's January 2026 final CDS guidance discussion of automation bias where users lack time for independent review.
• CI scope guards — build pipelines fail if image- or signal-processing code enters in-scope CDS paths (Criterion 1 firewall). See scripts/regulatory-checks.ts and scripts/lint-scope-boundary.ts.

ONC HTI-1 predictive-DSI transparency (31 source attributes for certified health IT) is the buyer-side mirror of this posture — ARKA publishes intended use, limitations, and maintenance commitments on public pages like this one and the Trust center.

Human oversight & monitoring

• Clinician retains the final call — every CDS card ends with the standard Non-Device CDS disclosure; the ordering clinician is responsible for the final decision.
• Override and feedback captured — accept, dismiss, and override actions are logged; Override / feedback rate is a first-class KPI on /outcomes and the validation dashboard.
• Near-miss review cadence — anonymized near-miss cards in the near-miss library feed QI review; weekly champion readouts during pilot shadow mode, then monthly with clinical governance.
• Subgroup and fairness monitoring — the validation dashboard includes a subgroup tab (age, sex, modality) for fairness gap review on synthetic cohorts; gaps require confirmation on real-world validation cohorts.
• Rule-library and model-weight change control — updates follow the documented Q-Sub change-control process with release notes to your team (open FDA Question 3 on /trust).
• Drift monitoring — monthly score-distribution review during pilots (weekly during shadow mode); quarterly at enterprise scale, aligned with the outcomes scorecard cadence.

What ARKA is not

• Not an FDA-cleared or -approved device — ARKA-CLIN is designed to meet Non-Device CDS criteria under FD&C Act §520(o)(1)(E) per FDA's January 2026 final CDS guidance; Q-Sub feedback is requested, not obtained.
• Not autonomous prior-authorization approval or denial — ARKA-INS supports documentation, eligibility, and transparency; UM staff and payers retain disposition authority.
• Not a diagnostic — ARKA does not acquire, process, or analyze medical images or physiological signals for diagnostic conclusions.
• Not a replacement for UM judgment — ARKA-INS is administrative-support software under §520(o)(1)(A), not clinical decision support for treatment selection.

Full ML model card (training data details, held-out metrics tables): /docs/model-card (shared under NDA).

Slide-ready summary

Copy the block below into committee decks or paste into governance submission templates. It prints on one page.