Agreement between automated and manual scoring of level 3 home sleep apnea devices
A single-center retrospective study finds that automated scoring of home sleep apnea tests systematically underestimates disease severity, misclassifying nearly half of moderate cases.
Lead
Home sleep apnea testing (HSAT) has become a cornerstone of obstructive sleep apnea (OSA) diagnosis, offering patients a convenient alternative to in-laboratory polysomnography. But a critical question has lingered: how reliable are the automated scoring algorithms built into these devices? A new study from Mubarak Al Kabeer Hospital in Kuwait provides a sobering answer.
Published in Sleep Medicine X, the retrospective analysis of 525 patients found that automated (computerized) scoring of Level 3 HSAT devices consistently underestimates the respiratory event index (REI) compared to manual scoring by trained sleep technicians. In some severity categories, up to 44.5% of patients with moderate OSA were downgraded to a milder classification — raising concerns about missed or delayed treatment.
What they found
The research team, led by Sulaiman Khadadah and colleagues, compared automated and manual scoring of Level 3 HSAT recordings — devices that typically monitor 3-4 channels including oximetry, airflow, respiratory effort, and heart rate, but not the full electroencephalography (EEG) of in-lab polysomnography.
The key findings were stark:
- Systematic underestimation. Automated REI values were consistently lower than manually derived REI values across the study population. Bland-Altman analysis confirmed a bias toward under-scoring, with limits of agreement wide enough to be clinically meaningful.
- Severity misclassification. When patients were stratified into standard OSA severity categories (none, mild, moderate, severe), automated scoring misclassified a substantial fraction. Most concerning was the 44.5% underestimation rate among moderate cases — patients whose disease was downgraded to mild or none by the algorithm.
- Hypopnea-driven gap. The discrepancy was most pronounced in the hypopnea index. Automated scoring detected significantly fewer hypopneas than manual scoring. Since hypopneas make up a large proportion of respiratory events in many patients, this single difference drove much of the overall REI underestimation.
- Threshold trouble. Misclassification clustered around the clinically important severity cutoffs — the boundaries between mild and moderate, and between moderate and severe. Patients whose true REI fell near these thresholds were most likely to be misclassified, because small scoring differences pushed them across the line.
The study used both “standard” and “extended” REI definitions, and the pattern held across both, strengthening the conclusion.
Why it matters
Obstructive sleep apnea affects an estimated 936 million adults worldwide, and the majority of cases remain undiagnosed. Home sleep apnea testing has been a critical tool for expanding access to diagnosis, particularly during and after the COVID-19 pandemic when in-lab testing faced capacity constraints.
However, the diagnostic value of HSAT depends on the accuracy of its scoring. If automated algorithms systematically underestimate severity, patients may be told their sleep apnea is “mild” when it is actually “moderate” — potentially forgoing treatments like positive airway pressure (PAP) therapy that could improve their sleep quality, cardiovascular risk, and daytime function.
The findings also highlight specific targets for improvement. Because the gap is driven primarily by hypopnea detection, refining algorithms to better identify these subtle events — characterized by drops in airflow accompanied by oxygen desaturation or arousal — could yield the greatest gains in accuracy. This is a concrete target for the next generation of AI-powered scoring tools.
Limits
As a single-center retrospective study, the results reflect the patient population and scoring practices at one institution. The Level 3 devices used may not represent all commercially available HSAT platforms. Manual scoring, while treated as the reference standard, also has inherent inter-scorer variability. The study did not compare either method against the gold standard of in-laboratory polysomnography with EEG, so the true accuracy of each approach remains unknown. Additionally, the study did not report outcomes — whether patients misclassified by automated scoring went on to have worse clinical outcomes — which would strengthen the clinical relevance of the findings.
Bottom line
Automated scoring of Level 3 home sleep apnea tests underestimates REI and OSA severity compared to manual technician scoring, particularly through under-detection of hypopneas. Clinicians should be aware of this bias when interpreting HSAT results, especially for patients whose automated REI falls near severity thresholds. The findings provide clear direction for improving automated and AI-driven scoring algorithms: better hypopnea detection is the single most impactful target.
Source
Khadadah S, Alanazi H, Saleh Y, Aljabri J, Elbagalaty MF, Ali A, Abdulsalam M. Agreement between automated and manual scoring of level 3 home sleep apnea devices: A single-center retrospective study. Sleep Medicine X. 2026 Jun 29;12:100193. DOI: 10.1016/j.sleepx.2026.100193. PMID: 42405233. PMCID: PMC13330509.