Diagnostic Accuracy of the Optical Coherence Tomography in Assessing Glaucoma Among Filipinos. Part 2: Optic Nerve Head and Retinal Nerve Fiber Layer Parameters

Glaucomatous optic nerve damage is a result of retinal ganglion cell (RGC) death with progressive loss of axons located in the retinal nerve fiber layer (RNFL). Several clinical studies showed that optic nerve head (ONH) damage and thinning of the RNFL occur earlier than the appearance of abnormalities in the visual field.1 Diagnostic modalities such as the optical coherence tomography (OCT) are primarily directed at demonstrating the presence of decreased thickness of the RNFL around the optic nerve head in glaucoma patients. The OCT is an accurate and reproducible method that measures and analyzes RNFL thickness and ONH parameters to help differentiate glaucomatous eyes from normal eyes.

This study determined the accuracy of the ONH and RNFL parameters using the Stratus OCT in the diagnosis of glaucoma among glaucoma suspects. It was a cross-sectional diagnostic validation study with a Phase 3 design as defined by Sackett.2 The Phase 3 diagnostic study design analyzed the ability of the OCT to assess patients that represented the target population for diagnostic testing using the Stratus OCT.

METHODOLOGY

This validation study was focused on the OCT parameters using the fast optic disc and fast RNFL protocols3-4 of the Stratus OCT machine as applied to glaucoma suspects.

A detailed description of the recruitment procedure, the inclusion and exclusion criteria, baseline data collection methods, randomization, sample size determination, determination of the reference standard, and ethical considerations for this study are reported in the first part of this Journal (Diagnostic accuracy of optical coherence tomography in assessing glaucoma among Filipinos. Part 1: Categorical outcomes based on a normative database).5 This report focused on the second objective of the study which evaluated the numerical results of the OCT.

Statistical Analysis

The baseline data, OCT numerical results, and the results of the expert assessment were analyzed using the SPSS version 16.0 software. For each of the OCT parameters, the t-test for difference of two means was performed and a nonparametric Welch-test in cases of unequal variances.

Below are the ONH and RNFL measurements that were retrieved from the printouts of the OCT study. Optic nerve head (ONH) analysis of 13 parameters:

Individual radial scan:
1. Rim area (vertical cross section) (mm2)
2. Average nerve width at disk (mm)
3. Vertical integrated rim area (VIRA) (mm2)
4. Horizontal integrated rim width (HIRW) (mm)
Six radial scan:
5. Disc area (mm2)
6. Disc diameter (mm)
7. Cup diameter (mm)
8. Rim length (mm)
9. Cup area (mm2)
10. Rim area (mm2)
11. Cup disc area ratio
12. Cup disc horizontal ratio
13. Cup disc vertical ratio

Retinal nerve fiber layer (RNFL) analysis of 25 parameters:
Over-all average
1. Average RNFL thickness (μm)
Quadrant averages
2. Superior RNFL average (μm)
3. Inferior RNFL average (μm)
4. Nasal RNFL average (μm)
5. Temporal RNFL average (μm)
Sector averages
6. 1 o’clock sector average (μm)
7. 2 o’clock sector average (μm)
8. 3 o’clock sector average (μm)
9. 4 o’clock sector average (μm)
10. 5 o’clock sector average (μm)
11. 6 o’clock sector average (μm)
12. 7 o’clock sector average (μm)
13. 8 o’clock sector average (μm)
14. 9 o’clock sector average (μm)
15. 10 o’clock sector average (μm)
16. 11 o’clock sector average (μm)
17. 12 o’clock sector average (μm)

Maximum thickness
18. Superior maximum (μm)
19. Inferior maximum (μm)
Comparisons

20. Smax/Imax
21. Smax/Tavg
22. Smax/Navg
23. Imax/Smax
24. Imax/Tavg
25. Max – Min (μm)

Receiver Operator Characteristic (ROC) curves were generated for each OCT ONH and RNFL parameters. The area under the ROC curve (AUC) was estimated at a 95% confidence level and was used to determine which of the top three parameters from the ONH and RNFL analyses have the best discriminant ability. Optimal cut-off point was computed using a statistical procedure patterned after a similar technique used by Ferreras and co-workers6 using the MedCalc Software Version 11.4.4 (downloadable from http:// www.medcalc.org/).

For the six best parameters based on the AUC for both the ONH and RNFL OCT scans, multi-level likelihood ratios were determined. The multi-level cut-off values were calibrated based on a posttest probability of at least 70% for a positive test result and 10% for a negative test result. Sensitivity, specificity, and likelihood ratios (LR) were estimated at 95% confidence interval. The diagnostic threshold was the specific point where a negative test results in a 10% posttest probability. The therapeutic threshold was the specific point where a positive test results in a 70% posttest probability.

RESULTS

The demographic data, baseline clinical data, and results of the reference standard determination were presented in part 1.5 Part 2 focused on the results of the analysis of the optic disc and RNFL parameters of the Stratus OCT.

ROC Curve Analysis: OCT Fast Optic Disc Parametersstrong>

For the different ONH parameters measured using the fast optic disc protocol, all parameters showed statistically significant differences between the glaucoma and the normal groups. With the exception of the disc area and disc diameter, all ONH parameters showed significant area under the curve (AUC) values in the ROC curve analysis. The various parameters measuring the rim and cup showed significant ability to discriminate between normal and glaucomatous eyes.

The best parameters from the optic nerve head analysis were the vertical integrated rim width (AUC 0.822), the cup:disc area ratio (0.816), and the horizontal integrated rim width (AUC 0.794).

Since the OCT optic nerve head analysis has no comparative normative database, there were no global indices available using the current version of its software. The cut-off values presented in Table 1 were derived from the ROC curve analysis using the MedCalc software.

ROC Curve Analysis: OCT Fast RNFL Parameters

All RNFL parameters showed statistically significant reductions in RNFL thickness in the glaucoma group as compared with the normal group (Table 2). A comparison of the reduction in thickness from the four quadrants showed that the inferior quadrant (mean diff: 34.66 μm) and the superior quadrant (mean diff: 29.58 μm) had greater reductions as compared with the nasal and temporal quadrants.

Figure 2. Effect of the cut-off determination using the MedCalc software. An average thickness cut-off value of 92.94 μm would result in a sensitivity of 76.5% and a false negative rate of 23.5%. The same would give a specificity of 75.8% and a false positive rate of 24.2%.

The ROC curves for selected RNFL parameters are shown in Figure 1. The parameters with the highest AUCs were average thickness (.827), superior average (.807), and inferior average (.804). The nasal and temporal quadrants had lower AUC values compared to the superior and inferior quadrants (Table 3). Among the clock sectors, the 11 o’clock sector (0.787) and the 7 o’clock sector (0.786) had the highest descriminant capacity. These sectors are known to be associated with the occurrence of inferior and superior arcuate scotomas which are charasteristics of glaucoma. The cut-off values for these parameters were based on an optimal sensitivity and specificity as derived from the MedCalc software. While most RNFL parameters showed highly significant differences in means between the two groups, this did not translate into high estimates of accuracy for any single parameter (Table 3). The best parameter, the average thickness,showed only a sensitivity of 76.47% and a specificity of 75.82% at an estimated cut-off value of 92.94 μm. The LR for a positive result using this cut-off was 3.16. This example illustrated that single cut-off point may not be useful for the OCT parameters.

Multi-level Likelihood Ratios for OCT ONH and RNFL Parameters

Multi-level likelihood ratios were derived using the actual data from this study (Table 4). The values for the likelihood ratio for a positive result were estimated and calibrated using a projected posttest probability of at least 70% for a positive test result which was identified as the therapeutic threshold. The values for the likelihood ratio for a negative result were estimated and calibrated using a projected posttest probability of 10% which was identified as the diagnostic threshold. In Table 4, the first interval showed the cut-off value and likelihood ratio for a positive result that would presumably rule in the disease. Therapeutic measures may be instituted since a positive result gives a high posttest probability of 70%. The 23% pretest probability is derived from the actual data for this study, but it may be higher or lower in other sample populations. The LR can then be used to derive the posttest probability for these other groups.

DISCUSSION

Validity of the OCT Parameters

This study assessed each parameter from the OCT optic nerve head and RNFL analyses. The AUC represents, in a single number, the diagnostic accuracy of a test wherein a value of 1 represents perfect discrimination, while a value of 0.5 represents random discrimination. OCT parameters with AUC values above 0.80 are generally considered to have good discriminating ability for a diagnostic test. Parameters with AUCs ranging from 0.70 to 0.80 are only fair, and those with AUCs below 0.7 are considered poor.

The average RNFL thickness remained the best parameter with the highest AUC, followed closely by the superior and inferior quadrant average. Among the clock-hour sectors, the best were the 7 o’clock and 11 o’clock sectors, followed closely by the 6 o’clock and 12 o’clock sectors. These OCT parameters had also been identified by previous studies as being the best for the diagnosis of glaucoma.

Kanamori identified the inferior quadrant and the 7 o’clock sector as the most sensitive for early glaucoma. They postulated that the thicker inferior quadrant is damaged early, with an accompanying superior field defect that is affected more than the inferior visual field.

Studies by Sihota identified the average thickness and the inferior quadrant as having the highest AUC.7 Ojima8 reported the average thickness, while Wollstein9 identified the rim area and the average thickness as the best OCT parameters. Medeiros10 identified the cup:disc area ratio as the best among the ONH parameters.

Budenz reported that the inferior quadrant, averagethickness, and superior quadrant had the largest AUCs at 0.971, 0.966, and 0.952 respectively. Medeiros credited the inferior quadrant with having the highest AUCs of 0.92 in patients with early to moderate glaucoma.

Our study showed that certain ONH parameters were as good as RNFL in discriminating glaucoma. Likelihood ratios for ONH parameters may be as useful to clinicians as those from the RNFL parameters. The VIRA (0.822), cup-to-disc area ratio (0.816), and the HIRW (0.794) had AUC values that were comparable with the average thickness, superior quadrant, and inferior quadrant.

The Low Predictive Ability of the OCT

The low predictive ability for each individual OCT parameter was due to overlapping distribution and wide range of values among the parameters, as exemplified by the frequency distribution of the two groups for the average thickness (Figure 3). The same distribution was seen in the other RNFL parameters. Thus, initial OCT measurements in a glaucoma suspect are not sufficient for concluding that the measured thickness is due to glaucoma. The marked variability may even be manifested before the disease begins, and two normal subjects could have widely disparate baseline RNFL thickness values and disc size. Calibrating Cut-off Points and Likelihood Ratios for Glaucoma

An ROC curve is a plot of the sensitivity versus 1-specificity in a diagnostic test, where the different points on the curve correspond to different cut-off points used to determine if the test results are positive. The ROC curve may be analyzed by a calibration process that would rely mainly on what threshold a clinician will be operating on. This would be in contrast to analyzing the ROC curve purely for its discriminatory power which would derive single cut-off points with the best sensitivity and specificity. There may be no accepted means of calibration. The effect of different cut-off points in the ROC curve calibration may be improved by using interval likelihood ratios, rather than a single cut-off point.

Likelihood ratio values greater than 10 or less than 0.1 usually generate large and conclusive changes from pretest to posttest probabilities. Values of 5-10 and 0.1-0.2 generate moderate shifts in pretest to posttest probabilities. Values of 2-5 and 0.2-0.5 generate small effects. Likelihood ratio values of 1 show insignificant effects.

The likelihood ratios shown in Table 4 presented three intervals; the top interval is used mainly for ruling in the disease, the lower interval is useful for ruling out the disease, and the middle interval is a middle ground at which the clinician may choose to repeat the tests at regular intervals and to monitor for progression of the suspect parameters. Clinically, glaucoma monitoring involves monitoring for increases in IOP, for enlargement of the cup-to-disc ratios, thinning of the neuroretinal rim, and progression of defects seen in the visual field. OCT as a monitoring tool would be most useful if an apparent deterioration is accompanied by one other structural evidence, such as those seen with the serial stereoscopic disc photos. Certainly, if a progression in a particular area, such as a clock hour sector, is accompanied by a progressive visual field defect on an area that corresponds to it functionally, the correlation between structure and function would help the clinician establish a decision on treatment.

Demonstration of progressive optic disc changes requires longitudinal follow-up and serial documentation of optic disc appearance. Patients with suspicious disc appearance who do not show any evidence of optic disc change or visual field loss during follow-up are usually considered as normal. It could be argued that some of these patients could still have damage, but that the follow-up time was insufficient to detect progression. This possibility cannot be completely discarded even if patients do not progress or develop functional loss after 9 years without any treatment.

The objectives of this study were to estimate test performance and the probability of disease in the glaucoma suspects. Test performance could be estimated based on the OCTs ability to diagnose the disease in comparison with a gold standard. The determination of AUCs and the estimation of posttest probabilities would serve to guide clinicians in estimating the probability of disease in these patients. Limitations

The use of likelihood ratios as a guide by clinicians reading OCT printouts can best be done when applied to settings similar to that of the study setting. The estimates derived from this study were based on a population of glaucoma suspects who were chosen and tested because of the presence of suspicious findings on initial ophthalmological evaluation. The study was done in a tertiary care setting, and subjects recruited to the study were referred by practicing ophthalmologists.

The estimates of test accuracy and validity may also be affected by the choice of the reference standard. The gold standard used in this study placed much importance on expert clinical assessment of the optic nerve head and the standard automatic perimetry (SAP). The criteria for a positive diagnosis of glaucoma include a combination of structural and functional evidence, such that a diagnosis of glaucoma could be made with certainty with or without an accompanying visual field defect. The visual field defect must also be typically glaucomatous. Global indices in SAP were less useful for this study.

There is no widely accepted gold standard for the diagnosis of glaucoma.15 For future research, it is essential that a gold standard for the definition of glaucoma be established. One possible gold standard would be the clinical evidence of progression of the glaucomatous damage.

In summary, a diagnosis of glaucoma should not be made based entirely on the results of the OCT. At the present time, the Stratus OCT cannot replace the gold standard of clinical assessment of structural and functional damage in the diagnosis of glaucoma. Because of its low sensitivity and high specificity for the diagnosis of glaucoma, the Stratus OCT may be used as a confirmatory test but not as a screening test.

The imaging information from the OCT should be considered as being complementary to other clinical measures. It is recommended that the multi-level likelihood ratios be used to guide clinicians on whether to start treatment or to do serial testing.

RNFL imaging allows the clinician to evaluate the rim thickness, the cup disc ratio, and the peripapillary RNFL thickness objectively. Repeat testing and follow-up measurements may be able to detect change over time.

It is recommended that further studies be done to validate the usefulness and applicability of the accuracy estimates reported in this study. The likelihood ratios may be validated in various settings,such as in longitudinal studies on a cohort of early glaucoma patients.

It is not hard to conceive of a time in the future when the prevailing reference standard for glaucoma may actually change. The OCT and the reference standard for glaucoma in this study may actually be measuring different parameters. SAP measures a physiological function while RNFL measures a structural function. But since the RNFL tissue damage appears earlier than the appearance of detectable visual field defects in most instances, the best diagnostic test might certainly be somewhere in the structural evaluation of the retinal nerve fiber layer.

Acknowledgements
We would like to thank Dr. Jose Ma. Martinez and Dr. Margaret Lat-Luna who served as the glaucoma experts for this study together with Dr. Tumbocon. Statistical analysis was performed in consultation with Dr. Jacinto Blas Mantaring and Prof. Cynthia Cordero from the Department of Clinical Epidemiology at the UP College of Medicine.

BIBLIOGRAPHY
1 American Academy of Ophthalmology. Ophthalmic Procedures Assessment: Optic nerve head and retinal nerve fiber layer analysis – a report by the American Academy of Ophthalmology. Ophthalmology 1999;106:1414-1424.
2 Sackett DL, Haynes RB. Evidence base of clinical diagnosis: The architecture of diagnostic research. Br Med J 2002; 321:539-541.
3 Medeiros FA, Vizzeri G, Zangwill LM, et al. Comparison of retinal nerve fiber layer and optic disc imaging for diagnosing glaucoma in patients suspected of having the disease. Ophthalmology 2008;115:1340–1346.
4 Kanamori A, Nakamura M, Escano MF. Evaluation of the glaucomatous damage on retinal nerve fiber layer thickness measured by optical coherence tomography. Am J Ophthalmol 2003;135:513–20.
5 Atienza NJ, Tumbocon JA. Diagnostic accuracy of the optical coherence tomography in assessing glaucoma among Filipinos. Part 1: Categorical outcomes based on a normative database. Philipp J Ophthalmol 2012;37:310.
6 Ferreras A, Pablo LE, Pajarin AB, et al. Logistic regression analysis for early glaucoma diagnosis using optical coherence tomography. Arch Ophthalmol 2008;126(4):465-470.
7 Sihota R, Sony P, Gupta V, et al. Diagnostic capability of optical coherence tomography in evaluating the degree of glaucomatous retinal nerve fiber damage. Invest Ophthalmol Vis Sci 2006;47:2006–2010.
8 Ojima T, Tanabe T, Hangai M. Measurement of retinal nerve fiber layer thickness and macular volume for glaucoma detection using optical coherence tomography. Jpn J Ophthalmol 2007;51:197–203.
9 Wollstein G, Ishikawa H, Wang J. Comparison of three optical coherence tomography scanning areas for detection of glaucomatous damage. Am J Ophthalmol 2005; 139:39– 43.
10 Medeiros FA, Zangwill LM, Bowd C. Evaluation of retinal nerve fiber layer, optic nerve head, and macular thickness measurements for glaucoma detection using optical coherence tomography. Am J Ophthalmol 2005;139:44–55.
11 Budenz DL, Michael A, Chang RT, et al. Sensitivity and specificity of the Stratus OCT for perimetric glaucoma. Ophthalmology 2005;112:3–9.
12 Mannasakorn A, Chaidaroon W, Ausayakhun S, et al.
Normative database of retinal nerve fiber layer and macular retinal thickness in a Thai population. Jpn J Ophthalmo­l 2008;52:450-456.
13 Zangwill LM and Bowd C. Retinal nerve fiber layer analysis in the diagnosis of glaucoma. Curr Opin Ophthalmol 2006;17:120–131.
14 Li G, Fansi AK, Boivin JF, et al. Screening for glaucoma in high risk populations using optical coherence tomography. Ophthalmology 2010;117:453-461.
15 American Academy of Ophthalmology. Ophthalmic Procedures Assessment: Optic nerve head and retinal nerve fiber layer analysis: a report by the American Academy of Ophthalmology. Ophthalmo­logy 2007;114:1937–1949.