Radiology
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH
QUICK SEARCH:   [advanced]


     

Published online before print December 21, 2005, 10.1148/radiol.2381041078

(Radiology 2005;238:480.)

A more recent version of this article appeared on December 1, 2005
This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow All Versions of this Article:
2381041078v1
238/2/480    most recent
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Heijenbrok-Kal, M. H.
Right arrow Articles by Hunink, M. G. M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Heijenbrok-Kal, M. H.
Right arrow Articles by Hunink, M. G. M.
© RSNA, 2005

Evidence-based Practice

Optimal Peak Systolic Velocity Threshold at Duplex US for Determining the Need for Carotid Endarterectomy: A Decision Analytic Approach1

Majanka H. Heijenbrok-Kal, PhD, Erik Buskens, MD, PhD, Paul J. Nederkoorn, MD, PhD, Yolanda van der Graaf, MD, PhD and M. G. Myriam Hunink, MD, PhD

1 From the Program for the Assessment of Radiological Technology (ART Program), Department of Epidemiology & Biostatistics, and Dept of Radiology, Erasmus MC-Univ Medical Center Rotterdam, EE21-40b, Dr Molewaterplein 50, 3015 GE Rotterdam, the Netherlands (M.H.H., M.G.M.H.); Department of Neurology, Academic Medical Center, Amsterdam, the Netherlands (P.J.N.); Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (E.B., Y.v.d.G.); and Department of Health Policy and Management, Harvard School of Public Health, Boston, Mass (M.G.M.H.). Received June 18, 2004; revision requested August 26; revision received March 7, 2005; accepted April 4; final version accepted May 4. Supported by program grant 904-66-091 from the Netherlands Organization for Scientific Research. Address correspondence to M.H.H. (e-mail: m.heijenbrok{at}erasmusmc.nl).


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 References
 
Purpose: To determine the optimal peak systolic velocity (PSV) threshold at duplex ultrasonography (US) required to establish the need for carotid endarterectomy in symptomatic patients on the basis of the long-term cost-effectiveness outcomes of diagnostic testing and subsequent treatment.

Materials and Methods: From January 1997 through January 2000, a prospective medical ethics committee–approved multicenter study was conducted. After giving informed consent, patients with amaurosis fugax, transient ischemic attack, or minor stroke who underwent duplex US and digital subtraction angiography were included in the study. Selective ipsilateral carotid angiograms were obtained in at least three planes. Arteries that were nearly or totally occluded at duplex US were excluded because the PSV cannot be reliably measured in these vessels. Receiver operating characteristic (ROC) curves were constructed for the diagnoses of 70%–99% and 50%–99% stenoses. Optimal likelihood ratios were calculated on the basis of lifetime costs and quality-adjusted life-years derived at cost-effectiveness analysis and the prevalence of disease. The associated optimal sensitivities, specificities, and PSV thresholds were derived from the ROC curves.

Results: In this clinical study, 350 patients were included. The nonoccluded arteries in a total of 236 patients were assessable for ROC analysis. For the diagnosis of 70%–99% stenosis, the optimal likelihood ratio was 0.21, which was associated with a PSV threshold of 220 cm/sec, a sensitivity of 97% (127 of 131 patients; 95% confidence interval [CI]: 94%, 100%), and a specificity of 48% (50 of 105 patients; 95% CI: 38%, 57%). For the diagnosis of 50%–99% stenosis, the optimal likelihood ratio was 0.38, which was associated with a PSV threshold of 180 cm/sec, a sensitivity of 95% (182 of 191 patients; 95% CI: 92%, 98%), and a specificity of 69% (31 of 45 patients; 95% CI: 55%, 82%).

Conclusion: On the basis of the lifetime outcomes of diagnostic testing and subsequent treatment, the optimal PSV thresholds for the diagnosis of 70%–99% and 50%–99% carotid artery stenoses in patients with amaurosis fugax, transient ischemic attack, or minor stroke were 220 cm/sec and 180 cm/sec, respectively.

© RSNA, 2005


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 References
 
The results of two large randomized trials—the North American Symptomatic Carotid Endarterectomy Trial (NASCET) and the European Carotid Surgery Trial—have shown carotid artery endarterectomy to yield a considerable benefit in patients with 70%–99% stenosis and a small benefit in patients with 50%–69% stenosis (1–6). In these trials, severe carotid artery stenosis was diagnosed by using cerebral angiography, the reference-standard examination. However, cerebral angiography is associated with a risk of mortality and morbidity and a marked financial cost. The routine use of cerebral angiography in patients with amaurosis fugax, transient ischemic attack, or minor stroke who are potential endarterectomy candidates is therefore undesirable. Consequently, many clinicians now use noninvasive examinations such as duplex ultrasonography (US), magnetic resonance (MR) angiography, and computed tomographic (CT) angiography to select patients for carotid endarterectomy (7). In a cost-effectiveness analysis, we recently found duplex US to be the optimal test strategy for selecting patients suitable for carotid endarterectomy (8).

Moreover, the Society of Radiologists in Ultrasound (SRU) has published recommendations for the interpretation of duplex US results in the diagnosis of internal carotid artery stenosis (7). These recommendations are based on the test characteristics and diagnostic accuracy of duplex US reported in the literature. In most of the diagnostic studies reviewed by the SRU, the optimal threshold of the peak systolic velocity (PSV) was based on the maximization of diagnostic accuracy. With maximization of accuracy as a criterion, one assumes that a false-negative test result has the same importance as a false-positive result. However, because the duplex US result is used to determine whether or not carotid endarterectomy will be performed, the consequences of missing a significant stenosis may be more or less favorable in terms of cost and/or effectiveness outcomes than the consequences of performing endarterectomy for a nonsignificant stenosis. Therefore, it is more clinically relevant to account for the costs and treatment effectiveness associated with false-positive and false-negative test results when selecting an optimal threshold for referring patients for endarterectomy.

Accordingly, the aim of our study was to determine the optimal PSV threshold at duplex US required to establish the need for carotid endarterectomy in symptomatic patients on the basis of the long-term cost-effectiveness outcomes of diagnostic testing and subsequent treatment.


   MATERIALS AND METHODS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
RESULTS
 DISCUSSION
 References
 
Study Population
From January 1997 through January 2000, a prospective diagnostic study was performed at two academic hospitals (University Medical Center Utrecht and Erasmus MC-University Medical Center Rotterdam) and one nonacademic hospital (Medical Spectrum Twente, Enschede, the Netherlands) (9). In this study, 350 patients with a mean age of 67 years (range, 39–88 years) were included and 76% (n = 266) of them were male (9). Two hundred forty-nine of these subjects were patients at University Medical Center Utrecht, 62 were patients at Erasmus MC-University Medical Center Rotterdam, and 39 were patients at Medical Spectrum Twente. After giving informed consent, patients with amaurosis fugax, transient ischemic attack, or minor stroke underwent carotid duplex US and carotid digital subtraction angiography (DSA) within a time frame of 4 weeks. The medical ethics committee at each hospital approved the study.

Duplex US of Carotid Arteries
In the majority of patients (n = 311 [89%]), duplex US was performed with an Ultramark 9 HDI or HDI 3000 (Advanced Technology Laboratories, Bothell, Wash) machine. For the 39 remaining patients, a Diasonics Master Series (GE Medical Systems, Milwaukee, Wis) unit was used. The Doppler angle was aligned to the jet and kept below 60°. The pulsed Doppler gate was positioned in the center of the common carotid artery, approximately 2 cm proximal to the carotid artery bifurcation, and a spectral waveform was obtained. Subsequently, the area with the most severe stenosis was located by using color Doppler US, and a Doppler spectral waveform was obtained at the point of the greatest mean frequency shift. From this spectrum, the PSV of the internal carotid artery was derived. Duplex US examinations were performed by qualified vascular technologists in the vascular laboratory of each hospital. The PSV was measured on a continuous scale, in centimeters per second, in the proximal region of the symptomatic internal carotid artery of each patient. A carotid artery was deemed to be symptomatic when the neurologic symptoms—specifically, amaurosis fugax, transient ischemic attack, or minor stroke—corresponded to the stenotic side. If no detectable blood flow was present, the patient was judged to have an occlusion—that is, 100% stenosis. Slow flow in combination with a visualized severe stenosis was defined as near occlusion. Arteries that were occluded or nearly occluded at US were excluded from receiver operating characteristic (ROC) analysis, because PSV cannot reliably be measured in these vessels.

DSA Examination
DSA was performed by using an Integris V3000 or Poly-I (Philips Medical Systems, Best, the Netherlands) angiographic unit with an image intensifier matrix of 1024 x 1024. In eight patients, an Angiostar Plus (Siemens Medical Systems, Forchheim, Germany) unit was used. Selective positioning of an intraarterial catheter in the common carotid artery was performed by using the Seldinger technique. From the carotid artery bifurcation, three projections (lateral, posteroanterior, and oblique) were acquired. The additional projections obtained at occasionally performed rotational DSA examinations were not used in this study. For each projection, 6 mL of iopromide (Ultravist 300; Schering, Berlin, Germany) was injected at a flow rate of 3 mL/sec or 9 mL of iomeprol (Iomeron 350; Altana Pharma, Hoofddorp, the Netherlands) was injected at a flow rate of 6 mL/sec.

The DSA results were read by two radiologists, each of whom had more than 5 years of experience. The observers were blinded to the patients' clinical information and duplex US results. They read the film hard copies of the DSA images. The stenosis percentage was measured according to NASCET criteria (1). Degree of stenosis was defined as the remaining lumen at the site of the stenosis divided by the normal lumen distal to the stenosis. Stenosis of 99% was defined as near occlusion. The maximal degree of stenosis seen on the three projections was used in the analyses. DSA was considered the standard of reference. Interobserver agreement regarding data obtained in a representative sample of 170 patients was determined by calculating {kappa} values.

Markov Model
The lifetime costs and quality-adjusted life-years (QALYs) rendered after obtaining true-positive, false-positive, true-negative, and false-negative duplex US results were derived by using a previously described Markov model (8). These values included the costs and utilities associated with the subsequent treatment, treatment complications, and progression of disease that ensued after these results were obtained. Several health states were modeled for the severity of neurologic disease (ie, transient ischemic attack, minor stroke, or major stroke as a presenting condition or a complication) in patients who initially had less than 50% stenosis, 50%–69% stenosis, or 70%–99% stenosis. Medical therapy, including aspirin treatment, was assumed to be the optimal treatment for patients with less than 50% stenosis. For patients with greater than 50% stenosis, two criteria for carotid endarterectomy referral were considered: 70%–99% stenosis and 50%–99% stenosis. We assumed that the associated costs and life expectancy after a carotid endarterectomy depended on the presenting symptoms (ie, transient ischemic attack or minor stroke) rather than on the patient's underlying stenosis category before surgery. Disease progression and death were modeled by simulating transitions to more severe health states during follow-up. The Markov model was constructed by using DATA Pro 11.0 software (TreeAge, Williamstown, Mass). The lifetime costs and effects were integrated into one measure, the net health benefit (NHB), which is derived by using the following equation:

(1)

where EQALY represents the lifetime effects in QALYs, C represents the lifetime costs, and TWTP is the amount that society is willing to pay (ie, willing-to-pay threshold) to save one QALY (10). With this formula, we assumed two threshold amounts that society would be willing to pay to save one QALY: $25 000 and $50 000 per QALY.

Recommended PSV Thresholds
The recommended PSV thresholds of 230 cm/sec for the diagnosis of 70%–99% stenosis and 125 cm/sec for the diagnosis of 50%–99% stenosis were applied to our study data (7). For both PSV thresholds, we calculated the associated sensitivity and specificity.

Optimal PSV Threshold at Statistical Analysis
We calculated the sensitivities and specificities associated with different PSV thresholds by using DSA as the reference-standard examination. We used two definitions of carotid artery disease, which corresponded to the two indications for referring patients for carotid endarterectomy: angiographically determined 70%–99% stenosis and 50%–99% stenosis. The cost-effectiveness outcomes of the Markov model for the 50%–69% stenosis category were combined with the results of either the 0%–49% category or the 70%–99% category (weighted for prevalence), depending on the indication for carotid endarterectomy. With use of ROC analysis, all combinations of sensitivity and specificity were plotted on a graph on which the y-axis represented sensitivity and the x-axis represented 1 – specificity (11). Smooth ROC curves were created by using summary ROC analysis methods (12–14). For each combination of sensitivity and specificity on the smooth curves, the result-specific likelihood ratio (LRRi)—that is, the probability (P) of a specific test result (Ri) (eg, a PSV of 220 cm/sec) in the group with the disease (DG) divided by the probability of that specific test result (eg, 220 cm/sec) in the group without the disease (NDG)—was calculated and was equal to the tangent, or the slope of the curve (15,16):

(2)

With the lifetime consequences of diagnostic testing and subsequent treatment taken into account, the optimal likelihood ratio (LRopt) depends on the prevalence of disease (p) and the ratio of the net loss due to false-positive test results compared with true-negative results (NHBTN – NHBFP) to the net loss due to false-negative results compared with true-positive results (NHBTP – NHBFN) (15–17):

(3)

The optimal combination of sensitivity and specificity (ie, the optimal operating point) was derived from the smooth ROC curve at the operating point where the result-specific likelihood ratio (Eq [2]) equaled the optimal likelihood ratio based on the Markov model (Eq [3]). Subsequently, we selected the PSV value that was observed to be closest to the operating point on the smooth curve and defined it as the optimal PSV threshold.

Sensitivity analysis of the prevalence of significant stenosis was performed to evaluate the effect of this prevalence on the optimal likelihood ratio, sensitivity, specificity, and PSV threshold. We used the SPSS 11.0 statistical software package (SPSS, Chicago, Ill) to perform summary ROC curve analysis and Excel 2000 (Microsoft, Redmond, Wash) to construct the ROC curves.


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
DISCUSSION
 References
 
DSA and Duplex US Results
The symptomatic carotid artery could be imaged and evaluated in 323 patients with DSA and in 330 patients with duplex US. The results of both examinations were available for 313 patients. Values were missing owing to the following reasons: Sometimes it was not feasible to perform both examinations before surgery, some patients withdrew from the study after having undergone one examination, and the examination was not always correctly performed according to our study protocol. Also, occasionally, the PSV was not measured when duplex US was performed. Finally, in seven patients, it was impossible to measure the degree of stenosis because of poor image quality and the poor reliability of the DSA findings. Interobserver agreement calculated on the basis of the DSA results for 170 patients was good ({kappa} = 0.79; 95% confidence interval [CI]: 0.74, 0.84).

In Table 1, the categorized DSA and duplex US results are cross tabulated. At DSA, 0%–49% stenosis was detected in 45 (14%) of 313 patients, 50%–69% stenosis was detected in 61 (20%), 70%–98% stenosis was detected in 128 (41%), near occlusion (ie, 99% stenosis) was detected in 16 (5%), and occlusion was detected in 63 (20%). Duplex US depicted occlusions in 61 patients. The sensitivity and specificity of the duplex US–based diagnosis of occlusion were 94% (59 of 63 patients; 95% CI: 88%, 100%) and 99% (248 of 250 patients; 95% CI: 98%, 100%), respectively. There was slow flow in combination with visualized severe stenosis at duplex US, indicating near occlusion, in 16 patients. In a total of 236 patients, arteries that were neither occluded nor nearly occluded at duplex US were assessable at ROC analysis.


View this table:
[in this window]
[in a new window]
  		Table 1. Categorized Internal Carotid Artery Measurements at Duplex US versus DSA

 
The data in Figure 1, in which absolute PSV measurements are plotted against angiographic degrees of stenosis, show that the relationship between PSV and stenosis degree is nonlinear. In Figure 1a and 1b, the indications for endarterectomy are set at minimal angiographic stenosis degrees of 70% and 50% (horizontal lines), respectively. Shifting the PSV threshold (vertical lines) to the left would result in an inversely related increase in the number of true-positive test results and a decrease in the number of true-negative test results. Thus, a decreased PSV threshold is associated with higher sensitivity and lower specificity. Conversely, shifting the PSV threshold upward would result in lower sensitivity and higher specificity.



View larger version (32K):
[in this window]
[in a new window]
  		Figure 1a: Scatterplots of absolute PSV measurements and angiographic stenosis measurements. PSV thresholds for referring patients with (a) 70%–99% stenosis (220 cm/sec) and (b) 50%–99% stenosis (180 cm/sec) for carotid endarterectomy are shown. Increasing the PSV threshold reduces the number of false-positive (FP) cases (higher specificity) at the expense of a higher number of false-negative (FN) cases (lower sensitivity). TN = true-negative cases, TP = true-positive cases.

 


View larger version (32K):
[in this window]
[in a new window]
  		Figure 1b: Scatterplots of absolute PSV measurements and angiographic stenosis measurements. PSV thresholds for referring patients with (a) 70%–99% stenosis (220 cm/sec) and (b) 50%–99% stenosis (180 cm/sec) for carotid endarterectomy are shown. Increasing the PSV threshold reduces the number of false-positive (FP) cases (higher specificity) at the expense of a higher number of false-negative (FN) cases (lower sensitivity). TN = true-negative cases, TP = true-positive cases.

 
Markov Model
Long-term outcomes of the cost-effectiveness analysis (Table 2) indicate that referring patients for carotid endarterectomy instead of medical therapy resulted in a loss in QALYs and an increase in costs for patients with 0%–49% stenosis. Performing endarterectomy facilitated a small gain in QALYs and a slight cost savings in patients with 50%–69% stenosis and a relatively large gain in QALYs and a large cost savings in patients with 70%–99% stenosis (Table 2).


View this table:
[in this window]
[in a new window]
  		Table 2. Lifetime Costs, QALYs, and NHBs for Base Cases with Positive and Negative Duplex US Results for Several Stenosis Categories

 
Recommended PSV Thresholds
The SRU has recommended using a PSV of 230 cm/sec for the diagnosis of 70%–99% stenosis and a PSV of 125 cm/sec for the diagnosis of 50%–99% stenosis (7). Applying the recommended threshold of 230 cm/sec to diagnose 70%–99% stenosis in our population would result in a sensitivity of 95% (125 of 131 patients; 95% CI: 92%, 99%) and a specificity of 51% (54 of 105 patients; 95% CI: 42%, 61%). If we applied a threshold of 125 cm/sec to detect 50%–99% stenosis in our study population, the sensitivity would be 99% (190 of 191 patients; 95% CI: 98%, 100%) and the specificity would be 36% (16 of 45 patients; 95% CI: 22%, 50%).

Optimal PSV Thresholds
For a referral for endarterectomy based on an indication of 70%–99% stenosis, the optimal result-specific likelihood ratio based on the Markov model (calculated by using Eq [3]) was 0.21. Figure 2a shows the observed ROC curve and the smooth ROC curve. Each dot on the observed curve represents the sensitivity and 1 – specificity of a specific PSV threshold for the angiographic diagnosis of 70%–99% stenosis—that is, for the eligibility for endarterectomy. The optimal likelihood ratio, or the slope of the ROC curve, is illustrated in this figure. The associated optimal sensitivity and specificity were 97% (127 of 131 patients; 95% CI: 94%, 100%) and 48% (50 of 105 patients; 95% CI: 38%, 57%), respectively. The optimal PSV threshold that corresponded to this sensitivity and specificity was 220 cm/sec (Fig 2a). The optimal PSV threshold is illustrated in Figure 1a also. The bottom right quadrant of this figure shows the large number of false-positive test results obtained by using the optimal PSV threshold compared with the top left quadrant, which contains a small number of false-negative test results obtained by using the optimal PSV threshold.



View larger version (29K):
[in this window]
[in a new window]
  		Figure 2a: Observed (ie, original) and smooth ROC curves for indications of (a) 70%–99% and (b) 50%–99% stenoses. The optimal likelihood ratio (LR)—that is, the slope of the ROC curve—derived at cost-effectiveness analysis is indicated with the associated sensitivity, specificity, and optimal PSV threshold. These data show that the optimal operating point on the ROC curve based on cost-effectiveness analysis results is different from the point on the curve that is closest to the upper left corner in ROC space, which is based on maximal accuracy.

 
When 50%–99% stenosis was used to indicate the need for endarterectomy, the optimal result-specific likelihood ratio was 0.38. The associated optimal sensitivity and specificity were 95% (182 of 191 patients; 95% CI: 92%, 98%) and 69% (31 of 45 patients; 95% CI: 55%, 82%), respectively (Fig 2b). The optimal PSV threshold that corresponded to this sensitivity and specificity was 180 cm/sec (Figs 1b, 2b).



View larger version (30K):
[in this window]
[in a new window]
  		Figure 2b: Observed (ie, original) and smooth ROC curves for indications of (a) 70%–99% and (b) 50%–99% stenoses. The optimal likelihood ratio (LR)—that is, the slope of the ROC curve—derived at cost-effectiveness analysis is indicated with the associated sensitivity, specificity, and optimal PSV threshold. These data show that the optimal operating point on the ROC curve based on cost-effectiveness analysis results is different from the point on the curve that is closest to the upper left corner in ROC space, which is based on maximal accuracy.

 
Changing the willingness-to-pay threshold from $25 000 to $50 000 per QALY hardly affected the optimal likelihood ratio and thus had no effect on the optimal PSV threshold. When a higher willingness-to-pay threshold is accepted, the NHB (Eq [1]) increases. The ratio of the difference in NHBs associated with specific test results, however, changed only minimally when a higher willingness-to-pay threshold was used (Eq [3]).

The results of the sensitivity analysis of the prevalence of disease are shown in Table 3. The data in this table demonstrate that when the prevalence of significant stenosis increases, the optimal likelihood ratio decreases and corresponds to higher sensitivity, lower specificity, and a decreased optimal PSV threshold. The data in Table 3 also show that the prevalence of significant disease (50%–99% stenosis) has to be very high (approximately 90%) before the threshold of 125 cm/sec recommended by the SRU becomes optimal.


View this table:
[in this window]
[in a new window]
  		Table 3. Influence of Prevalence of Significant Stenosis on Optimal Likelihood Ratio, Sensitivity, Specificity, and Optimal PSV Threshold

 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
References
 
We found that the optimal PSV threshold for selecting symptomatic patients for carotid endarterectomy was 220 cm/sec when the indication for endarterectomy was 70%–99% stenosis and 180 cm/sec when the indication was 50%–99% stenosis. The 220 cm/sec threshold is similar to the threshold recommended by the SRU (230 cm/sec for 70%–99% stenosis), but 180 cm/sec is higher than the recommended threshold (125 cm/sec for 50%–99% stenosis) (7).

The SRU based its recommendations on literature review findings. The studies included in that review were focused on optimizing accuracy, and the participating investigators did not take into account the variable effects of false-negative test results as opposed to the effects of false-positive test results. The results of our study show that referring a patient with nonsignificant stenosis (<50%) for endarterectomy is more harmful than missing a diagnosis of 50%–69% stenosis, and this finding explains the fairly high PSV threshold for discriminating 50%–99% stenosis from 0%–49% stenosis.

The natural history of the disease to be identified and the effectiveness of treatment have a major role in determining the importance of sensitivity and specificity. In cases of high-grade (>70%) carotid artery stenosis especially, undiagnosed disease is associated with high monetary and life expectancy costs (8). For the identification of patients with high-grade stenosis, duplex US criteria should be highly sensitive, yielding a minimal number of false-negative results, because these patients will have high degrees of morbidity and mortality if they are left untreated. The overall losses associated with missing a diagnosis in patients with 50%–99% stenosis are smaller, because the benefits associated with 50%–69% stenosis are smaller than those associated with 70%–99% stenosis. Balancing the losses and benefits led to a somewhat lower optimal sensitivity and a higher optimal specificity for the identification of 50%–99% stenosis compared with the identification of 70%–99% stenosis.

The prevalence of significant stenosis in the population being evaluated also has a critical role in defining the optimal test criterion. In our study population, the prevalence of 70%–99% stenosis was 46% and the prevalence of 50%–99% stenoses was 66%. Sensitivity analysis revealed that if the prevalence of significant stenosis were lower, the derived slope would be steeper (Eq [3]) and the optimal cutoff point on the ROC curve would shift to the left, implying lower sensitivity, higher specificity, and thus a higher PSV threshold.

Increasing the societal willingness-to-pay threshold from $25 000 to $50 000 per QALY did not influence the choice of the optimal PSV threshold in our study. However, this is not always the case: The optimal likelihood ratio can either increase or decrease—depending on the proportions of true and false results in a particular test situation—and thus lead to a higher or lower threshold when the willingness-to-pay threshold is increased.

Various methods of determining the optimal diagnostic cutoff point on the ROC curve have been reported in the literature. Some investigators have used the point on the curve that is closest to the upper left corner in the ROC space as the optimal cutoff point (18); others have used the cutoff point associated with a likelihood ratio of 1 (19). Other methods include selecting the Q point—that point where sensitivity equals specificity (20); maximizing accuracy, or the sum of the sensitivity and specificity values (21,22); and accepting a preset level of sensitivity (or specificity) and determining the corresponding specificity (or sensitivity) (23). All of these methods help to minimize the number of false-positive and false-negative test results. However, none of these methods involves taking into account the prevalence of disease or the consequences—in terms of costs and quality of life—of correctly or incorrectly classifying a test result as positive or negative.

The decision analytic approach of determining the optimal diagnostic threshold that we used was described many years ago (15). Published practical applications of this method, however, are scarce. In publications on carotid artery disease, we found only two studies in which the investigators based their optimal test criteria on patient outcomes rather than test accuracy (24,25). Wilterdink and co-workers (24) based their criteria on the 2-year mortality and morbidity rates associated with severe stenosis treated medically versus surgically, as reported in the NASCET. They observed a slope, or optimal likelihood ratio, of 0.09, which is more lenient than the slope that we observed and implies higher sensitivity and lower specificity. They used duplex US to select patients for angiography, whereas we used duplex US to select patients for endarterectomy.

The harm of performing unnecessary angiography in false-positive cases in the Wilterdink et al study (24) was much smaller than the harm of performing unnecessary endarterectomy in the current study. In both the Wilterdink et al study and the current study, however, the harm of a false-negative test result was the same: missing the opportunity to reduce the probability of a carotid event by means of endarterectomy. These factors explain the lower slope. Furthermore, we used updated results from the NASCET study, which show a small but significant benefit for patients with 50%–69% stenosis; we integrated both costs and effects on life expectancy; and we modeled lifetime outcomes.

Kuntz et al (25) chose the PSV cutoff point that minimized the probability of stroke at 2 years for symptomatic patients. They observed an optimal PSV threshold of 229 cm/sec at one laboratory and 340 cm/sec at another laboratory. They did not report the optimal likelihood ratio, or the slope of the ROC curve. We did not evaluate the potential differences between hospitals. Hospitals that do have data on angiography and PSV measurements, however, could construct their own ROC curve and determine their own optimal threshold by using the optimal slope derived from our decision analysis.

Our study population consisted of patients with transient ischemic attack, minor stroke, or amaurosis fugax. The optimal PSV threshold may vary among patients with different symptoms of carotid disease. We were unable to calculate optimal PSV thresholds for each subgroup of patients because the subgroups were too small for us to derive accurate ROC curves with enough cutoff values that were evenly spread along the curve. Moreover, optimal PSV thresholds may be different for asymptomatic patients, because the associated costs and consequences of testing and treatment for these patients differ from those for symptomatic patients.

For the estimation of ROC curves, the test variable needs to be a measure that continuously increases or decreases with the severity of disease. The PSV, however, increases with the severity of stenosis but decreases with near occlusion and is absent with total occlusion; therefore, we had to exclude these conditions. This was justified in the cases of total occlusion, because carotid endarterectomy is not indicated for totally occluded arteries. Duplex US was most accurate for the diagnosis of the total occlusions but not very accurate for the diagnosis of the near occlusions. Overall, excluding the near and total occlusions did not result in an overestimation of the diagnostic accuracy of duplex US.

Currently, new noninvasive tests that yield excellent images of the carotid arteries, such as CT angiography and MR angiography, are available (26). With the availability of these examinations, preoperative carotid angiography is hardly needed anymore. However, these new examinations are not yet accepted as reference-standard tests; therefore, we used carotid angiography as the reference-standard test in our ROC analyses.

Furthermore, we took into account no velocity parameters except the PSV. In several studies, it has been shown that the PSV in the internal carotid artery is the best single velocity parameter for quantifying stenosis (27,28). In clinical practice, multiple clinical parameters could be combined to account for differences in patients and to decide which patients should undergo carotid endarterectomy.

In our analyses, we used a decision analytic model to calculate the harms and benefits of diagnostic testing and subsequent treatment. This model is a simplification of reality. The input variables came from multiple sources, assumptions had to be made, and uncertainty surrounded the input variables. Thus, when new therapies become available, the model may need to be changed. For example, our calculations are based on the use of aspirin as the optimal medical therapy for symptomatic patients with less than 50% stenosis, in accordance with the NASCET and European Carotid Surgery Trial protocols. However, these patients are increasingly being treated with statins and clopidogrel also. Therefore, the diagnostic thresholds need to be updated when data on the long-term effects of statins and clopidogrel in these patients become available.

In conclusion, on the basis of the lifetime consequences of diagnostic testing and subsequent treatment, the optimal PSV threshold was 220 cm/sec for the diagnosis of 70%–99% carotid artery stenosis and 180 cm/sec for the diagnosis of 50%–99% stenosis in patients with amaurosis fugax, transient ischemic attack, or minor stroke when duplex US was used to refer patients for carotid endarterectomy.


   FOOTNOTES
 

Abbreviations: CI = confidence interval • DSA = digital subtraction angiography • NASCET = North American Symptomatic Carotid Endarterectomy Trial • NHB = net health benefit • PSV = peak systolic velocity • QALY = quality-adjusted life-year • ROC = receiver operating characteristic • SRU = Society of Radiologists in Ultrasound

Authors stated no financial relationship to disclose.

Author contributions: Guarantor of integrity of entire study, M.G.M.H.; study concepts, all authors; study design, M.G.M.H.; literature research, M.H.H.; clinical studies, P.J.N., Y.v.d.G.; data acquisition, P.J.N., E.B.; data analysis/interpretation, M.H.H., M.G.M.H.; statistical analysis, M.H.H., M.G.M.H.; manuscript preparation, M.H.H.; manuscript definition of intellectual content, editing, revision/review, and final version approval, all authors


   References
TOP
 ABSTRACT
 INTRODUCTION
 MATERIALS AND METHODS
 RESULTS
 DISCUSSION
 References
 

  1. North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N Engl J Med 1991;325:445–453.[Abstract]
  2. European Carotid Surgery Trialists' Collaborative Group. MRC European Carotid Surgery Trial: interim results for symptomatic patients with severe (70–99%) or with mild (0–29%) carotid stenosis. Lancet 1991;337:1235–1243.[CrossRef][Medline]
  3. Barnett HJ, Taylor DW, Eliasziw M, et al; for the North American Symptomatic Carotid Endarterectomy Trial Collaborators. Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis. N Engl J Med 1998;339:1415–1425.
  4. Randomised trial of endarterectomy for recently symptomatic carotid stenosis: final results of the MRC European Carotid Surgery Trial (ECST). Lancet 1998;351:1379–1387.[CrossRef][Medline]
  5. Rothwell PM, Gutnikov SA, Warlow CP. Reanalysis of the final results of the European Carotid Surgery Trial. Stroke 2003;34:514–523.[Abstract/Free Full Text]
  6. Rothwell PM, Eliasziw M, Gutnikov SA, et al. Analysis of pooled data from the randomised controlled trials of endarterectomy for symptomatic carotid stenosis. Lancet 2003;361:107–116.[CrossRef][Medline]
  7. Grant EG, Benson CB, Moneta GL, et al. Carotid artery stenosis: gray-scale and Doppler US diagnosis—Society of Radiologists in Ultrasound Consensus conference. Radiology 2003;229:340–346.[Abstract/Free Full Text]
  8. Buskens E, Nederkoorn PJ, Buijs-Van Der Woude T, et al. Imaging of carotid arteries in symptomatic patients: cost-effectiveness of diagnostic strategies. Radiology 2004;233:101–112.[Abstract/Free Full Text]
  9. Nederkoorn PJ, Mali WP, Eikelboom BC, et al. Preoperative diagnosis of carotid artery stenosis: accuracy of noninvasive testing. Stroke 2002;33:2003–2008.[Abstract/Free Full Text]
  10. Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making 1998;18(2 suppl):S68–S80.[Abstract/Free Full Text]
  11. Obuchowski NA. Receiver operating characteristic curves and their use in radiology. Radiology 2003;229:3–8.[Abstract/Free Full Text]
  12. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med 1993;12:1293–1316.[Medline]
  13. Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Making 1993;13:313–321.[Abstract/Free Full Text]
  14. Irwig L, Tosteson AN, Gatsonis C, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med 1994;120:667–676.[Abstract/Free Full Text]
  15. Metz CE. Basic principles of ROC analysis. Semin Nucl Med 1978;8:283–298.[Medline]
  16. Hunink MG, Glasziou PP, Siegel JE, et al. Multiple test results. In: Decision making in health and medicine: integrating evidence and values. Cambridge, United Kingdom: Cambridge University Press, 2001.
  17. Phelps CE, Mushlin AI. Focusing technology assessment using medical decision theory. Med Decis Making 1988;8:279–289.[Abstract/Free Full Text]
  18. Criswell BK, Langsfeld M, Tullis MJ, Marek J. Evaluating institutional variability of duplex scanning in the detection of carotid artery stenosis. Am J Surg 1998;176:591–597.[CrossRef][Medline]
  19. Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med 2000;45:23–41.[CrossRef][Medline]
  20. Westwood ME, Kelly S, Berry E, et al. Use of magnetic resonance angiography to select candidates with recently symptomatic carotid stenosis for surgery: systematic review. BMJ 2002;324:198.[Abstract/Free Full Text]
  21. Carsten CG 3rd, Elmore JR, Franklin DP, Thomas DD, Mordan F, Wood GC. Use of limited color-flow duplex for a carotid screening project. Am J Surg 1999;178:173–176.[CrossRef][Medline]
  22. AbuRahma AF, Robinson PA, Strickler DL, Alberts S, Young L. Proposed new duplex classification for threshold stenoses used in various symptomatic and asymptomatic carotid endarterectomy trials. Ann Vasc Surg 1998;12:349–358.[CrossRef][Medline]
  23. Fillinger MF, Baker RJ Jr, Zwolak RM, et al. Carotid duplex criteria for a 60% or greater angiographic stenosis: variation according to equipment. J Vasc Surg 1996;24:856–864.[CrossRef][Medline]
  24. Wilterdink JL, Feldmann E, Easton JD, Ward R. Performance of carotid ultrasound in evaluating candidates for carotid endarterectomy is optimized by an approach based on clinical outcome rather than accuracy. Stroke 1996;27:1094–1098.[Abstract/Free Full Text]
  25. Kuntz KM, Polak JF, Whittemore AD, Skillman JJ, Kent KC. Duplex ultrasound criteria for the identification of carotid stenosis should be laboratory specific. Stroke 1997;28:597–602.[Abstract/Free Full Text]
  26. Randoux B, Marro B, Koskas F, et al. Carotid artery stenosis: prospective comparison of CT, three-dimensional gadolinium-enhanced MR, and conventional angiography. Radiology 2001;220:179–185.[Abstract/Free Full Text]
  27. Hunink MG, Polak JF, Barlan MM, O'Leary DH. Detection and quantification of carotid artery stenosis: efficacy of various Doppler velocity parameters. AJR Am J Roentgenol 1993;160:619–625.[Abstract/Free Full Text]
  28. Elgersma OE, van Leersum M, Buijs PC, et al. Changes over time in optimal duplex threshold for the identification of patients eligible for carotid endarterectomy. Stroke 1998;29:2352–2356.[Abstract/Free Full Text]




This Article
Right arrow Abstract Freely available
Right arrow Figures Only
Right arrow All Versions of this Article:
2381041078v1
238/2/480    most recent
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Heijenbrok-Kal, M. H.
Right arrow Articles by Hunink, M. G. M.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Heijenbrok-Kal, M. H.
Right arrow Articles by Hunink, M. G. M.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH
RADIOLOGY RADIOGRAPHICS RSNA JOURNALS ONLINE