| 摘要: |
| 目的 使用多阅片者多病例（MRMC）设计评价人工智能（AI）辅助阅片影像诊断试验的临床效能，以期为影像诊断试验的临床评价提供科学依据。方法 采用影像诊断试验中广泛应用的MRMC设计，详细阐述了MRMC设计中Obuchowski-Rockette（OR）法的模型构建及其检验方法。实例研究共收集了3家医院200例受试者的CT影像资料，其中133例为肋骨骨折患者，68例为非肋骨骨折患者，由3位阅片医师对所有CT影像进行判读。分析在2种阅片方式（医师＋AI辅助阅片、医师独立阅片）下肋骨骨折检出的AUC值、灵敏度和特异度的差异。结果 AI辅助阅片组的AUC值为0.958，医师独立阅片组的AUC值为0.902，两组AUC值差异有统计学意义（P＜0.001）。AI辅助阅片组总体的灵敏度为0.970，特异度为0.946；医师独立阅片组的灵敏度为0.838，特异度为0.966；两组灵敏度差值为0.131（95% CI 0.091～0.171），特异度差值为－0.020（95% CI －0.059～0.020），说明AI辅助阅片与医师独立阅片的灵敏度差异有统计学意义而特异度差异无统计学意义。两组的阳性似然比均大于10，阴性似然比均小于0.2，阳性预测值都接近1，说明AI辅助阅片影像诊断试验的诊断准确性高。结论 AI辅助阅片在提高诊断效能方面有显著优势，不仅可以提高肋骨骨折诊断的准确性和检出率，还能提高医师工作效率，优化医院服务。 |
| 关键词: 人工智能 多阅片者多病例设计 Obuchowski-Rockette法 肋骨骨折 诊断准确性 |
| DOI：10.16781/j.CN31-2187/R.20240775 |
| 投稿时间：2024-11-14修订日期：2024-12-26 |
| 基金项目:上海市卫生健康委员会新兴交叉领域研究专项（2022JC011）. |
|
| Application of multi-reader multi-case design in evaluating artificial intelligence-assisted imaging diagnostic trials |
| WAN Huiqin1,XIANG Man1,PAN Zhemin1,QIN Yingyi2,HE Qian2,HE Jia1,2* |
(1. School of Medicine, Tongji University, Shanghai 200092, China;
2. Department of Military Health Statistics, Faculty of Medical Services, Naval Medical University (Second Military Medical University), Shanghai 200433, China
*Corresponding author) |
| Abstract: |
| Objective To evaluate the clinical efficacy of artificial intelligence (AI)-assisted imaging diagnostic trials using multi-reader multi-case (MRMC) design, so as to provide a scientific basis for clinical evaluation of imaging diagnostic trials. Methods The MRMC design, widely used in imaging diagnostic trials, was adopted in this study. The Obuchowski-Rockette (OR) method of MRMC design was detailed, including model construction and test methods. A case study was conducted, collecting imaging data of 200 subjects from 3 hospitals, with 133 cases of rib fractures and 68 cases of non-rib fractures. Three radiologists reviewed all CT images of the subjects. The area under curve (AUC) value, sensitivity and specificity in detecting rib fractures between 2 reading modalities (radiologists with AI assistance vs radiologists reading independently) were compared. Results The AI-assisted reading group had an AUC value of 0.958, while the radiologist-independent reading group had an AUC value of 0.902, showing a significant difference (P＜0.001). The overall sensitivity and specificity of the AI-assisted reading group were 0.970 and 0.946, respectively; while the sensitivity and specificity of the radiologist-independent reading group were 0.838 and 0.966, respectively. The difference of sensitivity between groups was 0.131 (95% confidence interval ［CL］ 0.091-0.171), and the difference of specificity was －0.020 (95% CI －0.059-0.020), indicating a significant difference in sensitivity but not in specificity between AI-assisted and radiologist-independent reading groups. Both groups had positive likelihood ratios (＋LR) greater than 10 and negative likelihood ratios (－LR) less than 0.2, with positive predictive values approaching 1, suggesting that the diagnostic accuracy of the AI-assisted imaging diagnostic trials was high. Conclusion The AI-assisted reading method demonstrates a significant advantage in enhancing diagnostic efficiency, not only improving the diagnostic accuracy and detection rate of rib fractures, but also improving the work efficiency of radiologists and optimizing hospital services. |
| Key words: artificial intelligence multi-reader multi-case design Obuchowski-Rockette method rib fractures diagnostic accuracy |
