Abstract:Objective To evaluate the clinical efficacy of artificial intelligence (AI)-assisted imaging diagnostic trials using multi-reader multi-case (MRMC) design, so as to provide a scientific basis for clinical evaluation of imaging diagnostic trials. Methods The MRMC design, widely used in imaging diagnostic trials, was adopted in this study. The Obuchowski-Rockette (OR) method of MRMC design was detailed, including model construction and test methods. A case study was conducted, collecting imaging data of 200 subjects from 3 hospitals, with 133 cases of rib fractures and 68 cases of non-rib fractures. Three radiologists reviewed all CT images of the subjects. The area under curve (AUC) value, sensitivity and specificity in detecting rib fractures between 2 reading modalities (radiologists with AI assistance vs radiologists reading independently) were compared. Results The AI-assisted reading group had an AUC value of 0.958, while the radiologist-independent reading group had an AUC value of 0.902, showing a significant difference (P<0.001). The overall sensitivity and specificity of the AI-assisted reading group were 0.970 and 0.946, respectively; while the sensitivity and specificity of the radiologist-independent reading group were 0.838 and 0.966, respectively. The difference of sensitivity between groups was 0.131 (95% confidence interval [CL] 0.091-0.171), and the difference of specificity was -0.020 (95% CI -0.059-0.020), indicating a significant difference in sensitivity but not in specificity between AI-assisted and radiologist-independent reading groups. Both groups had positive likelihood ratios (+LR) greater than 10 and negative likelihood ratios (-LR) less than 0.2, with positive predictive values approaching 1, suggesting that the diagnostic accuracy of the AI-assisted imaging diagnostic trials was high. Conclusion The AI-assisted reading method demonstrates a significant advantage in enhancing diagnostic efficiency, not only improving the diagnostic accuracy and detection rate of rib fractures, but also improving the work efficiency of radiologists and optimizing hospital services.