Inter- and Intra-Observer Reliability Among Retinopathy of Prematurity (ROP) Screeners
Kristine Corpus, MD, Jubaida Aquino, MD, Macario Reandelar Jr., MD, MSPH, Retinopathy of Prematurity Working Group
Objective
(1) To determine the inter and intra-observer reliability in diagnosing ROP in terms of the stage, zone, and presence of plus disease among local ROP screeners involved in the ROP Working Group; and (2) to determine the inter-observer reliability between 2 groups of subspecialties – retina specialists and pediatric ophthalmologists.
Methods
This is a prospective observational study that analyzed the inter- and intra-observer reliability in describing ROP in 3 key observations: stage, zone, and presence of plus disease. This study utilized a test with 32 sets of fundus images from 27 cases, five of which were repeated. Images from previously photographed infants with and without ROP were collated into a downloadable powerpoint test and tested against retina specialists and pediatric ophthalmologists of the ROP Working Group. Outcome measures included presence of variability in ROP diagnosis in terms of the stage, zone, and presence of plus disease among screeners, and reliability coefficient (intra-class coefficient or ICC) in 2 levels: (1) individual and 2-group inter-observer reliability, and (2) intra-observer reliability.
Results
There were 11 respondents: 5 retina specialists and 6 pediatric ophthalmologists. Seven (46%) reported prior experience with RetCam image review. There was high inter-observer reliability (ICC 1.0) in the staging of ROP, but poor reliability in the identification of zone (ICC 0.3) and plus disease (ICC 0.5). The group of retina specialists and pediatric ophthalmologists scored high reliability for diagnosis of stage (ICC 1.0 vs 0.9) and plus disease (ICC 0.9 vs 0.9), while both showed poor reliability in the identification of zone (ICC 0.5 vs 0.4). Majority had high intra-observer reliability with regard to the stage (55%) and zone (73%) of ROP and most (73%) had acceptable intra-observer reliability in identifying plus disease. None of the respondents had poor intra-observer reliability.
Conclusions
The diagnosis of the stage of ROP was consistently reliable for both inter- and intra-observer parameters. However, identification of zone of ROP and plus disease were sources of significant discrepancies.
Keywords: ROP, Retinopathy of prematurity, Screening, Variability, ROP grading