HM Medical Clinic

 

Survey.erasmusmc.nl

Research and Reporting Methods Annals of Internal Medicine
Net Reclassification Improvement: Computation, Interpretation,
and Controversies
A Literature Review and Clinician's Guide
Maarten J.G. Leening, MD, MSc; Moniek M. Vedder, MSc; Jacqueline C.M. Witteman, PhD; Michael J. Pencina, PhD;
and Ewout W. Steyerberg, PhD
The net reclassification improvement (NRI) is an increasingly pop- ing NRI analysis is proposed: Detail and motivate the methods used ular measure for evaluating improvements in risk predictions. This for computation of the NRI, use clinically meaningful risk cutoffs for article details a review of 67 publications in high-impact general the category-based NRI, report both NRI components, address is- clinical journals that considered the NRI. Incomplete reporting of sues of calibration, and do not interpret the overall NRI as a NRI methods, incorrect calculation, and common misinterpretations percentage of the study population reclassified. Promising NRI find- were found. To aid improved applications of the NRI, the article ings need to be followed with decision analytic or formal cost- elaborates on several aspects of the computation and interpretation in various settings. Limitations and controversies are discussed, in-cluding the effect of miscalibration of prediction models, the useof the continuous NRI and "clinical NRI," and the relation with Ann Intern Med. 2014;160:122-131.
decision analytic measures. A systematic approach toward present- For author affiliations, see end of text.
Since the introduction of the term risk factor more than when these models are used. The subsequent changes in
50 years ago in this journal (1), many such factors have risk classification can be quantified by the net reclassifica- been identified. Risk factors have been incorporated into tion improvement (NRI) (15). Risk reclassification analysis statistical models to predict occurrence of disease, to more with the NRI has become popular: More than 1000 pub- adequately diagnose patients, and to predict outcomes after lications have cited the 2008 article that introduced the disease has been diagnosed. A substantial number of clini- NRI (15). However, reporting of the methods used is of cal guidelines have incorporated risk prediction models to heterogeneous quality (16), and misconceptions are com- aid clinicians in everyday decision making in various fields mon in interpreting the NRI (17).
of medicine, including cardiology, oncology, and respira- In this article, we aim to provide a systematic assess- tory medicine (2– 8).
ment of the reporting practices in analyses involving the Many markers, such as biomarkers, genetic factors, NRI and address some controversies relating to its use and and imaging results, have been proposed to improve these interpretation. We also make recommendations on how to prediction models. In the past 3 decades, the most com- report and interpret the NRI (18).
monly used measure to quantify these improvements hasbeen the change in the c-statistic, also known as the areaunder the receiver-operating characteristic curve (AUC).
OVERVIEW OF CURRENT REPORTING
Studies have emphasized the limitations of the AUC, in- Literature Search and Data Extraction
cluding the difficulty in interpreting the usually small We systematically collected studies that computed the changes in this statistic and the relation of the magnitude NRI or discussed results from NRI analysis. We used the of improvement to the performance of the baseline model Thomson Reuters Web of Knowledge (version 5.9) to (9 –12). A more relevant criterion may be to assess whether identify all publications that cited 1 of 4 methodological the addition of the marker to an existing model will influ- articles by Pencina and colleagues (15, 19 –21) or a meth- ence clinical practice (13), which is the case if the newly odological review on reclassification measures by Cook and predicted risk crosses a clinically meaningful threshold for Ridker (22). The search was last updated on 23 April 2013 an individual. This has led to the introduction of the con- and yielded 1250 unique citations (Appendix Figure 1,
cept of risk reclassification (14), which involves cross- available at We selected all 67 citations tabulating categories of predicted risk for 2 models— in the 4 general clinical journals with the highest impact usually one with the new marker under study and the other factors (New England Journal of Medicine, The Lancet, Jour- without it—to see how persons are classified differently nal of the American Medical Association, and Annals of In-
ternal Medicine) (22– 88) for data extraction (Appendix
Tables 1 and 2, available at Our ratio-
nale was that these articles may be expected to have broad
impact and be used as examples for others.
Two evaluators independently extracted data from the publications. Cases on which the evaluators disagreed werediscussed with a third evaluator to reach consensus. All 122 2014 American College of Physicians
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Net Reclassification Improvement: Literature Review and Clinician's Guide Research and Reporting Methods publications were searched for NRI calculations or results.
lines in 4 (11%) instances (Table 2). For outcomes other
If these were found, we checked which version of the NRI than atherosclerotic cardiovascular disease, the rationale for was used: the category-based NRI (15) or the continuous the risk categorization could not be traced in 10 of 12 (category-free) NRI (20) (Table 1). Next, we reviewed all
instances. Another 8 studies on the prediction of various articles to determine whether risk categories corresponding manifestations of cardiovascular disease used cutoffs for the to diagnostic or treatment thresholds from clinical guide- NRI that are the subject of ongoing debate (28, 60, 70, 89, lines were used to evaluate the category-based NRI or 90)—for example, a 10-year risk cutoff of 6% (rather than whether other categorization was justified. We determined 10%) for low risk for coronary heart disease. Fourteen which NRI components were reported: solely the overall publications applied cutoffs for coronary risk stratification NRI, or the event NRI and the nonevent NRI (Table 1).
to broader definitions of cardiovascular disease (Appendix
Moreover, we categorized studies that reported estimates of Table 1).
the overall NRI on the basis of whether they reported it as Among 38 prospective studies that calculated the NRI, a unitless statistic or a percentage.
30 (79%) clearly reported the time horizon at which the risk predictions were evaluated. In 7 of 30 (23%) instances The predominant reason for citing one of the meth- where both predicted horizon and observed follow-up were odological articles was the computation of NRI estimates detailed, we could infer that the authors studied a pre- (n ⫽ 39) (Table 2). In 2 (5%) articles, only the continuous
dicted horizon beyond the observed follow-up time (Table
NRI was computed. In 5 articles, the NRI was used to 2). We identified another 7 studies that used events occur-
compare 2 different models instead of the nested addition ring beyond the predicted horizon in the reclassification of 1 or more new risk markers to a simpler model.
Of the 37 articles that computed category-based NRI Nearly all studies reported the overall NRI. Only 11 results, 34 (92%) detailed the cutoffs for the risk categories (28%) articles presented its components—the event NRI chosen. The number of risk categories defined in the com- and the nonevent NRI—in the results section. However, putation of the NRI varied between 2 and 6, with 3 being 25 (68%) presented reclassification tables stratified for the most common number (Appendix Table 1). These risk
events and nonevents (Table 2), which allowed for com-
categories were justified in the text, by references, or both putation of both NRI components by a knowledgeable ways in 15 (41%) instances and fully matched clinically reader. By combining the components presented in the text meaningful categories with clear implications from guide- and the reclassification tables, we identified 29 (74%) stud- Table 1. Formulas and Interpretation of the NRI
Formula and Interpretation
Pr(up event) ⫺ Pr(down event) ⫽ (number of events classified up ⫺ number of events classified down)/number of eventsThe net percentage of persons with the event of interest correctly classified upwardCan be interpreted as a percentage with a range of ⫺100% to 100%† Pr(down nonevent) ⫺ Pr(up nonevent) ⫽ (number of nonevents classified down ⫺ number of nonevents classified up)/number of The net percentage of persons without the event of interest correctly classified downwardCan be interpreted as a percentage with a range of ⫺100% to 100%† [Pr(up event) ⫺ Pr(down event)] ⫹ [Pr(down nonevent) ⫺ Pr(up nonevent)] ⫽ event NRI ⫹ nonevent NRIThe sum of the net percentages of correctly reclassified persons with and without the event of interest; this statistic is implicitly weighted for the event rate and cannot be interpreted as a percentage Theoretical range is –2 to 2 Pr(higher event) ⫺ Pr(lower event) ⫽ (number of events with increased predicted risk ⫺ number of events with decreased predicted risk)/number of events The net percentage of persons with the event of interest correctly assigned a higher predicted riskCan be interpreted as a percentage with a range of ⫺100% to 100%† Pr(lower nonevent) ⫺ Pr(higher nonevent) ⫽ (number of nonevents with decreased predicted risk ⫺ number of nonevents with increased predicted risk)/number of nonevents The net percentage of persons without the event of interest correctly assigned a lower predicted riskCan be interpreted as a percentage with a range of ⫺100% to 100%† [Pr(higher event) ⫺ Pr(lower event)] ⫹ [Pr(lower nonevent) ⫺ Pr(higher nonevent)] ⫽ event NRI ⫹ nonevent NRIThe sum of the net percentages of persons with and without the event of interest correctly assigned a different predicted risk; this statistic is implicitly weighted for the event rate and cannot be interpreted as a percentage Theoretical range is ⫺2 to 2 NRI ⫽ net reclassification improvement; Pr ⫽ probability.
* Assumes that clinically meaningful categories of predicted risk can be defined.
† Negative percentages are interpreted as a worsening in risk classification (i.e., the number of incorrectly reclassified events [or nonevents] exceeds the number of correctlyreclassified events [or nonevents]).
‡ Does not consider any categorization.
21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 123
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Research and Reporting Methods Net Reclassification Improvement: Literature Review and Clinician's Guide used to calculate the predicted risks should be clear. Be- Table 2. Results From the Literature Review on Reporting of
cause virtually every prospective study has some loss to follow-up, it is important to adequately handle observa-tions with incomplete follow-up in the analysis. In our Reporting of NRI Feature
Studies, n (%)
review, we found that studies published shortly after the Reason for citing methodological article on NRI
introduction of the NRI often did not report how incom- Claimed to have calculated NRI Discussed NRI results from previous analysis plete follow-up was handled. Some studies classified cen- Suggested alternative methods for quantifying sored observations as nonevents ("naive extrapolation") or predictive abilities Computed other (non-NRI) measures elaborated excluded persons with incomplete follow-up. Better meth- on in this article ods have been proposed to limit loss of useful information,including Kaplan–Meier estimates of the expected number Only continuous (category-free) NRI computed of events and nonevents ("prospective NRI") (20, 78) and Categorization for computing NRI detailed inverse-probability weighting (91). Similarly, not every Categorization for computing NRI justified in text study has sufficient follow-up available for the predicted Reference given for NRI categorization Categorization for computing NRI corresponded to time horizons used in clinical guidelines (for example, 10- diagnostic or therapeutic implications in clinical year risk for coronary heart disease [89]). In the articles we reviewed, authors made various attempts to overcome this Time horizon and follow-up
problem, such as using Weibull extrapolation (48, 53), ad- Predicted horizon detailed justing the predicted risk cutoffs by the ratio of actual to Observed follow-up detailed (mean, median, or desired follow-up (24), or extrapolating the observed rates Predicted time horizon longer than observed on the Kaplan–Meier survival estimates to the predicted time horizon for presentation purposes (22).
Event NRI and nonevent NRI in text or tables The NRI was introduced with the example of the Reclassification table for main findings added value of high-density lipoprotein cholesterol level to coronary risk prediction in the Framingham Heart Study Reported as a percentage (15). Current clinical guidelines on primary prevention of Interpreted as a percentage or proportion cardiovascular disease recommend clear cutoffs for initia- NRI ⫽ net reclassification improvement.
tion of statin treatment (2, 3, 89, 90). These recommen- * Of all 67 publications included in the literature review.
† Of 39 studies that calculated the NRI.
dations are supported by cost-effectiveness analyses. The ‡ Of 37 studies that calculated the category-based NRI.
NRI captures the change in a person's predicted risk that § Of 38 prospective studies that calculated the NRI.
㛳 Of 30 prospective studies that calculated the NRI and detailed the predicted crosses one of such cutoffs and thus translates into a clin- horizon and follow-up.
ically meaningful change in treatment recommendations.
¶ Table 1 provides more details.
** Of 36 studies that reported the overall NRI.
Our review of the literature confirms the findings of Tzoulaki and colleagues: Selected risk cutoffs are generally ies with information on the event NRI and nonevent NRI poorly motivated and rarely correspond to therapeutic im- presented for at least 1 reclassification analysis. Of note, 1 plications. Both shortcomings have been shown to yield study claimed to have calculated the NRI, but no such significantly higher NRI estimates (16, 81). In some cases, results could be traced. Another study presented P values the existing clinical cutoffs may result in limited reclassifi- but no point estimates of the NRI.
cation. For example, in a study of a population at very low Of the 36 studies presenting estimates of the overall risk for cardiovascular disease, only a small number of par- NRI, 24 (67%) expressed it as a percentage (Table 2).
ticipants would be considered to be at high risk; therefore, Eight (22%) articles in our review interpreted the overall few will cross the recommended risk thresholds after the NRI as a percentage or proportion of the entire study pop- addition of a new marker (92). Using the existing cutoffs ulation that was correctly reclassified or used similar word- illustrates the limited utility of a new marker in real-life ing, such as interpreting an overall NRI of 0.29 as application to such a low-risk population. Choosing a pri- " . . 29% of patients were correctly reclassified . . " (17, ori clinically meaningful cutoffs has been frequently em- phasized (15, 16, 19, 20, 60, 63, 81, 92–98). In addition,the estimates of the NRI and its components increase with NRI COMPUTATION, COMPONENTS, AND
the number of categories (95, 99). Limiting analysis to clinically meaningful categories will forestall authors from Predicted Time Horizons and Follow-up
presenting results from the cutoffs with the highest magni- When prospective data are involved, such as cardiovas- tude of NRI in their data. Moreover, consistent use of cular events occurring during follow-up, the time horizon cutoffs enhances comparability of results on the same 124 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Net Reclassification Improvement: Literature Review and Clinician's Guide Research and Reporting Methods markers between studies provided that the same outcome cases, which implies artificial weighting by the investigators definition and time horizons are used.
(43). This should not lead to different estimates in magni- Although many risk prediction algorithms are de- tude of the NRI compared with results derived from a full scribed in the medical literature, a limited number of clin- cohort provided that the cases and controls are randomly ical guidelines outside the field of cardiology explicitly rec- selected (20, 104). However, difficulties arise when selected ommend risk thresholds for use in clinical practice. In the controls are not representative of the entire underlying sub- fields where meaningful cutoffs are lacking or evolving, set they were drawn from, as in the case when matching on various options have been suggested to overcome this prob- certain risk factors (even as simple as age and sex) is done lem. Each has its own caveats. First, in some cases, classi- (104 –106). This can be overcome by weighting for the fication thresholds exist for related outcomes. For example, inverse of the sampling probability for cases and controls a 20% 10-year risk for "hard coronary heart disease" cor- responds to a 25% 10-year risk for "total coronary heartdisease" (100). In these situations, a conversion factor Components and Interpretation
based on the ratio of event rates—in this example, a ratio Although the article that introduced the NRI recom- of 1.25— can be used to translate cutoffs from one appli- mended reporting the components of the overall NRI (15), cation to another. Such conversion assumes that the asso- we noticed in our review that a limited number of studies ciated clinical implications are similar for the different out- did so. The components are easier to interpret than the come definitions, which may not always be true. For combined number: When only 1 cutoff is being evaluated, example, the protective effect of statins on the occurrence the event NRI equals the improvement in sensitivity and of cardiovascular manifestations other than coronary heart the nonevent NRI equals the improvement in specificity disease, such as heart failure, may be less (101). Similarly, (15). The NRI components then express the net percent- conversion factors can be used to define risk cutoffs for ages of persons with or without events correctly reclassified different predicted time horizons (for example, 30- vs. 10- (Table 1). Negative percentages for the components are
year risks [102]). In the absence of published conversion interpreted as a net worsening in risk classification. The factors, the data under study can be examined to define the overall NRI is the sum of these 2 underlying components; relative occurrence of the outcomes. Second, some re- as a result, an identical point estimate of this statistic may searchers have suggested defining risk categories based on have different interpretations depending on its components the event rate. A cutoff equal to the event rate would be (62, 93). Large positive values of the event NRI indicate used for binary classification, and cutoffs equal to half the that the investigated marker aids in the detection of per- event rate, the event rate, and twice the event rate would be sons with the outcome of interest. This enables clinicians used when more than 2 categories are desired (99, 103).
to initiate targeted treatment and thereby prevent events.
Such cutoffs, however, have no direct clinical interpreta- On the other hand, an overall NRI driven by the nonevent tion. The appropriateness of risk cutoffs should be related NRI indicates the marker's property of correctly decreasing to the anticipated use of the prediction model. As an ex- risk estimates for nonevents and is thus useful for reducing ample, myocardial infarction risk thresholds for a model overtreatment. However, such markers will have limited used to select patients with chest pain for early discharge contribution to decreasing the burden of disease. This il- from an emergency department will be much lower than lustrates the difficulty of interpreting the overall NRI with- those for a model used to identify patients with chest pain out knowledge of its components (107). Although it is who will benefit from early invasive coronary angiography.
tempting to do so, the overall NRI cannot be interpreted as Third, the continuous NRI was introduced as an alterna- the "net percentage of persons correctly reclassified" in a tive in the absence of any categorization (Table 1) (20).
straightforward manner (48) because of the implicit However, it does not quantify the clinical impact of risk weighting by the event rate: The overall NRI is the sum of reclassification (see Limitations and Controversies). The re- 2 fractions with different denominators (the number of lation between cutoffs and the risk distribution in the data events and nonevents) (17). Such misinterpretations may can be elegantly visualized in reclassification graphs with have contributed to the popularity of the overall NRI, superimposed cutoffs (Appendix Figure 2, available at
which therefore should not be presented as a percentage but as a unitless statistic (17). Moreover, the componentsof the overall NRI may be reasonably well-interpretable, whereas their sum is less so because of the implicit weight- Because of cost and feasibility, the predictive value of ing related to the event rate (the costs of misclassification new biomarkers is often studied in subsets of persons with are assumed to be proportional to the odds of nonevents) events and nonevents from larger prospective studies, espe- (Table 1) (108).
cially when the event rates are low. The NRI can be used in As with most summary statistics, the NRI should not both cohort studies and (nested) case– control studies (20).
be interpreted on its own but in the context of comple- In the latter, the researcher determines the ratio of events mentary statistical measures. If a marker is not associated (cases) to nonevents (controls) by selective oversampling of with the outcome or does not yield an increase in the 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 125
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Research and Reporting Methods Net Reclassification Improvement: Literature Review and Clinician's Guide AUC, a positive NRI should not be expected (94). In rare is which one leads to better classification (which relates to instances where this does occur, random chance or differ- both discrimination and calibration of the models). On the ences in calibration between the models are the most likely other hand, when the focus is primarily on the potential of causes. Also, presenting reclassification tables (in tabular or a new marker, the improvements in discrimination and graphical form) will aid in the broader interpretation of subsequent risk reclassification that it can induce are of summarized reclassification statistics (Appendix Figure 2
primary interest.
and Appendix Table 3, available at
The continuous NRI was originally proposed to over- LIMITATIONS AND CONTROVERSIES
come the problem of selecting categories in applications where they do not naturally exist (20). It does not require Unlike such rank-based statistics as the AUC, the NRI any risk categorization and considers all changes in pre- is affected by miscalibration of a model (that is, the average dicted risk for all events and nonevents. This has several predicted risk is not close to the event rate) (108 –110).
consequences. First, most changes in predicted risk do not Systematic miscalibration does not occur when the perfor- translate into changes in clinical management; for example, mance of models is assessed on the same data set that was a middle-aged woman whose 10-year predicted coronary used to develop them but is often present when prediction risk doubles from 1% to 2% will probably not be treated models are validated in other populations. A well- differently (92, 119). Therefore, the interpretation of the recognized example of this phenomenon is the application continuous NRI is different from that of the category- of the Framingham cardiovascular risk models to European based NRI (Table 1) (11). Second, when the addition of a
populations (111–113). When performing a head-to-head normally distributed marker is considered, the continuous comparison between a Framingham function (using the NRI is less affected by the performance of the baseline published coefficients and baseline hazard) and a new risk model and can therefore be seen as a rescaling of the mea- function developed from the data under study, one might sures of association (for example, an odds ratio of 1.65 per find an NRI that favors the new model and no difference SD corresponds to a continuous NRI of 0.395) (11, 21).
in the AUCs (114, 115). This discrepancy can be avoided Consequently, the continuous NRI is often positive for by deriving both the reference model and the model in- relatively weak markers (11). Moreover, it is strongly af- cluding the marker under investigation from the same data fected by miscalibration, especially in the setting of exter- set that is used to compute the NRIs or by recalibrating nal validation (110).
both models in case of independent validation (116).
As such, the continuous NRI is less suitable for head- The traditional Hosmer–Lemeshow goodness-of-fit to-head comparisons of competing models unless these test is strongly dependent on the sample size of the study models have been developed from the same data or are (117). Therefore, calibration might better be assessed correctly calibrated. The most appealing application of the graphically in a plot with predicted risks on the horizontal continuous NRI comes in quantifying the effect of an axis and observed event rates on the vertical axis, as in added predictor in settings where the distributions of other Koller and colleagues' example (54). For perfectly cali- risk factors may not be representative of the population brated models, the plot forms a diagonal line where the (120). For example, when the same marker for coronary observed event rates equal the predicted risks. Such graphs risk prediction is evaluated in 2 populations, one with wide can show systematic underestimation or overestimation as and the other with narrow age ranges, the conclusions well as issues of overfitting (which can be quantified using about its usefulness might be different if based on the in- the calibration slope [118]).
crement in AUC (12). The continuous NRI, however,would give a consistent message and is therefore marker- Classification or Reclassification?
descriptive rather than model-descriptive. Furthermore, its Some researchers have argued that before addressing magnitude should be assessed on its own scale (11) and the issue of reclassification, one should first focus on risk should not be compared with that of the category-based classification and examine the margins of a reclassification table (43). Accordingly, examining reclassification is usefulonly to the extent to which it quantifies change in the size Clinical NRI
of these margins. This might be of particular relevance in Reclassification measures, including the NRI, can be head-to-head comparisons of nonnested models with sub- used to evaluate markers in specific subgroups of the study stantial reclassification (that is, if the 2 models have low population defined by the reference model. Specifically, the correlation). In this case, as in the example shown by added value of new risk markers may be of greater impor- Koller and colleagues (54), knowing how many persons are tance in persons with a risk categorization that has more classified in the clinically relevant subgroups is of greater uncertainty about the clinical implications (for example, interest than the exact reclassification within the inner cells persons at intermediate risk for coronary heart disease [33, of the table (93, 96). Therefore, when choosing between 48, 62, 72, 73, 86]). This "clinical NRI" (121), however, competing models for clinical practice, the main question has been found to be biased because it does not take into 126 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Net Reclassification Improvement: Literature Review and Clinician's Guide Research and Reporting Methods account incorrect reclassification from other risk categories Table 3. Recommendations for Reporting the NRI
into the intermediate-risk category (62). Adding randomlygenerated noninformative markers to existing predictionmodels leads to positive clinical NRIs more frequently than expected on the basis of chance (99, 122). A method for Specify the type of NRI computed in the methods correcting this systematic overestimation has been pub- section of the manuscript (category-based lished (122).
and/or continuous NRI).
Specify the horizon of risk prediction if the NRI Decision Analytic Measures
was computed for prognostic evaluations (e.g., The overall NRI implicitly weights for the event rate, Describe how censored observations (e.g., p, with 1/p and 1/(1 ⫺ p) serving as costs for false-negative persons lost to follow-up before the specified results (events classified downward) and false-positive re- horizon) were handled.
Use the event status at the predicted time sults (nonevents classified upward), respectively (108, 123).
horizon and ignore events occurring beyond However, a different weighting of false-positive and false- the predicted time horizon (e.g., when negative results is often more clinically appropriate (98).
predicting 10-y risk for CHD, considerparticipants with a myocardial infarction This can readily be incorporated in a weighted version of occurring after 10 y of follow-up as the NRI if the event NRI and nonevent NRI are presented For category-based NRI, the categorization should separately or when a reclassification table is provided (20, ideally have clear consequences in clinical 124). In its broadest form, the weighted NRI can be inter- preted as the average savings (for example, in dollars or When possible, give references to formal clinical guidelines used to define the risk categories for quality-adjusted life-years) per person resulting from using the computation of the NRI.
the new model instead of the old one (20).
If alternative cutoffs were used, clearly motivate The weighted NRI is a decision analytic measure and is mathematically a transformation of changes in net ben- efit and relative utility (124). These measures use the Report the NRIs for events and nonevents harm– benefit ratio to define an optimum decision thresh- Reclassification tables stratified for persons with old for binary classification as high risk versus low risk and without the event of interest are
informative beyond the NRI (e.g., Appendix
(125). The harm– benefit ratio also defines the weights of Table 3).
true-positive and false-positive classifications to calculate a The event and nonevent NRIs can be presented single summary measure (124 –126). However, the use of as percentages. However, the overall NRI hasno units and should therefore not be presented such decision analytic measures is limited by the fact that as a percentage (Table 1).
weights for harms and benefits are not firmly established in Provide information on the calibration of the most fields of medicine (126), although a range of decision models being compared.
thresholds can be considered in a sensitivity analysis with visualization in a "decision curve" (127).
The components of the overall NRI can be interpreted as a net percentage of the number The nonweighted category-based NRI analysis is re- of persons with or without events. However, garded as an early-stage analysis in the evaluation of new the overall NRI should not be interpreted as a markers or prediction models. For assessment of the poten- net percentage of the study populationcorrectly reclassified.
tial clinical utility of promising markers, decision analytic Do not draw strong comparative conclusions approaches are needed in the next step, after the NRI anal- based on direct comparisons of NRIs obtainedin different populations or using different yses but before a full formal cost-effectiveness analysis that outcomes or cutoffs.
incorporates changes in costs and clinical outcomes inmore detail (13).
CHD ⫽ coronary heart disease; NRI ⫽ net reclassification improvement.
all circumstances. Also, the sum of the NRI components In our literature review, we encountered several com- should not be interpreted as a percentage. If authors choose mon flaws in the presentation and interpretation of the to present the category-based NRI, they should discuss the NRI and insufficient documentation of the computational implied costs of misclassification by the event rate. The methods. On the basis of our observations, we make the cutoffs selected for the NRI analyses should preferably following recommendations for clinical research (18) match risk thresholds that have clear clinical implications (Table 3).
or can be motivated on clinical grounds. In general, the Clearly defining which type of NRI is used is essential category-based NRI is directly applicable in settings where because their applicability and relevance vary substantially.
meaningful risk categories exist and models are well- The most appropriate NRI type and cut points depend on calibrated. If either of these conditions is not satisfied, one several factors, as discussed in this review. We recommend must carefully determine what information the NRI offers separate reporting of the NRI for events and nonevents in and whether it can be interpreted meaningfully. Using cut- 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 127
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Research and Reporting Methods Net Reclassification Improvement: Literature Review and Clinician's Guide offs that have no direct clinical meaning impedes the in- 3. Perk J, De Backer G, Gohlke H, Graham I, Reiner Z, Verschuren M, et al;
terpretation of the category-based NRI. Several methods European Association for Cardiovascular Prevention & Rehabilitation
(EACPR). European Guidelines on cardiovascular disease prevention in clinical
have been proposed to define cut points in situations where practice (version 2012). The Fifth Joint Task Force of the European Society of meaningful thresholds do not exist, but each has its own Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical caveats. Presenting graphical displays similar to a decision Practice (constituted by representatives of nine societies and by invited experts).
curve (127) for a range of cutoffs could be considered as Eur Heart J. 2012;33:1635-701. [PMID: 22555213] an alternative. The continuous NRI can be recommended 4. Hamm CW, Bassand JP, Agewall S, Bax J, Boersma E, Bueno H, et al; ESC
Committee for Practice Guidelines. ESC Guidelines for the management of
in only a few settings, including those where the primary acute coronary syndromes in patients presenting without persistent ST-segment focus is on the strength of the marker rather than model elevation: The Task Force for the management of acute coronary syndromes performance. Authors must be careful not to overinterpret (ACS) in patients presenting without persistent ST-segment elevation of the Eu- the magnitude of the continuous NRI, which is usually ropean Society of Cardiology (ESC). Eur Heart J. 2011;32:2999-3054. [PMID: much larger than that of the category-based NRI, and must 21873419]
5. Visvanathan K, Chlebowski RT, Hurley P, Col NF, Ropka M, Collyar D,
ascertain that the models are well-calibrated. Finally, for et al; American Society of Clinical Oncology. American Society of Clinical
mathematical reasons, we recommend against calculating P Oncology clinical practice guideline update on the use of pharmacologic inter- values for any of the forms of the NRI when the contribu- ventions including tamoxifen, raloxifene, and aromatase inhibition for breast can- tion of a new marker is being evaluated (128, 129). In- cer risk reduction. J Clin Oncol. 2009;27:3235-58. [PMID: 19470930]
6. Munshi NC, Anderson KC, Bergsagel PL, Shaughnessy J, Palumbo A, Durie
stead, after a marker has been shown to be statistically B, et al; International Myeloma Workshop Consensus Panel 2. Consensus
significantly associated with the outcome, only CIs for the recommendations for risk stratification in multiple myeloma: report of the Inter- NRI should be presented.
national Myeloma Workshop Consensus Panel 2. Blood. 2011;117:4696-700.
Our recommendations are meant to improve com- pleteness, transparency, and clinical relevance of research 7. Worth LJ, Lingaratnam S, Taylor A, Hayward AM, Morrissey S, Cooney J,
et al; Australian Consensus Guidelines 2011 Steering Committee. Use of risk
involving risk reclassification. However, because the scien- stratification to guide ambulatory management of neutropenic fever. Australian tific debate on the NRI and related performance measures Consensus Guidelines 2011 Steering Committee. Intern Med J. 2011;41:82-9.
is ongoing, our recommendations may be subject to ad- vances or additions in the future.
8. Bates SM, Jaeschke R, Stevens SM, Goodacre S, Wells PS, Stevenson MD,
et al; American College of Chest Physicians. Diagnosis of DVT: Antithrombotic
Therapy and Prevention of Thrombosis, 9th ed: American College of Chest
From Erasmus MC - University Medical Center Rotterdam, Rotterdam, Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141:e351S- the Netherlands, and Duke University, Durham, North Carolina.
418S. [PMID: 22315267]
9. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of
Grant Support: By the Netherlands Organisation for Health Research
the odds ratio in gauging the performance of a diagnostic, prognostic, or screen- and Development (ZonMw) and the Netherlands Organisation for Sci- ing marker. Am J Epidemiol. 2004;159:882-90. [PMID: 15105181] entific Research (NWO) (ZonMw HTA grant 80-82500-98-10208; Vici 10. Cook NR. Use and misuse of the receiver operating characteristic curve in
grant 918-76-619) and the Center for Translational Molecular Medicine risk prediction. Circulation. 2007;115:928-35. [PMID: 17309939] (PCMM project grant). The funding sources had no role in the design or 11. Pencina MJ, D'Agostino RB, Pencina KM, Janssens ACJW, Greenland P.
conduct of the study; the collection, management, analysis, or interpre- Interpreting incremental value of markers added to risk prediction models. Am J tation of the data; or the preparation, review, or approval of the Epidemiol. 2012;176:473-81. [PMID: 22875755] 12. Austin PC, Steyerberg EW. Predictive accuracy of risk factors and markers: a
simulation study of the effect of novel markers on different performance measures
for logistic regression models. Stat Med. 2013;32:661-72. [PMID: 22961910]
Potential Conflicts of Interest: Disclosures can be viewed at
13. Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind
MSV, et al; American Heart Association Expert Panel on Subclinical Athero-
sclerotic Diseases and Emerging Risk Factors and the Stroke Council. Criteria
for evaluation of novel markers of cardiovascular risk: a scientific statement from
Requests for Single Reprints: Maarten J.G. Leening, MD, MSc, De-
the American Heart Association. Circulation. 2009;119:2408-16. [PMID: partment of Epidemiology, Erasmus MC - University Medical Center Rotterdam, PO Box 2040, 3000 CA Rotterdam, the Netherlands; 14. Cook NR, Buring JE, Ridker PM. The effect of including C-reactive protein
in cardiovascular risk prediction models for women. Ann Intern Med. 2006;145:
21-9. [PMID: 16818925]
15. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating
Current author addresses and author contributions are available at the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157-72. [PMID: 17569110]
16. Tzoulaki I, Liberopoulos G, Ioannidis JPA. Use of reclassification for assess-
ment of improved prediction: an empirical evaluation. Int J Epidemiol. 2011;40:
1094-105. [PMID: 21325392] 1. Kannel WB, Dawber TR, Kagan A, Revotskie N, Stokes J 3rd. Factors of risk
17. Leening MJG, Steyerberg EW. Fibrosis and mortality in patients with di-
in the development of coronary heart disease—six year follow-up experience. The lated cardiomyopathy [Letter]. JAMA. 2013;309:2547-8. [PMID: 23800926] Framingham Study. Ann Intern Med. 1961;55:33-50. [PMID: 13751193] 18. Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, Rock F,
2. Expert Panel on Detection, Evaluation, and Treatment of High Blood Cho-
et al. Issues in methodological research: perspectives from researchers and com-
lesterol in Adults. Executive Summary of The Third Report of The National
missioners. Health Technol Assess. 2001;5:1-57. [PMID: 11368832] Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, 19. Pencina MJ, D'Agostino RB, Vasan RS. Statistical methods for assessment
And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel of added usefulness of new biomarkers. Clin Chem Lab Med. 2010;48:1703-11.
III). JAMA. 2001;285:2486-97. [PMID: 11368702] 128 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Net Reclassification Improvement: Literature Review and Clinician's Guide Research and Reporting Methods 20. Pencina MJ, D'Agostino RB Sr, Steyerberg EW. Extensions of net reclassi-
fication improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11-21. [PMID: 21204120] 40. Helfand M, Buckley DI, Freeman M, Fu R, Rogers K, Fleming C, et al.
21. Pencina MJ, D'Agostino RB Sr, Demler OV. Novel metrics for evaluating
Emerging risk factors for coronary heart disease: a summary of systematic reviews improvement in discrimination: net reclassification and integrated discrimination conducted for the U.S. Preventive Services Task Force. Ann Intern Med. 2009; improvement for normal variables and nested models. Stat Med. 2012;31:101- 151:496-507. [PMID: 19805772] 13. [PMID: 22147389] 41. Hingorani AD, Psaty BM. Primary prevention of cardiovascular disease: time
22. Cook NR, Ridker PM. Advances in measuring the effect of individual pre-
to get more or less personal? JAMA. 2009;302:2144-5. [PMID: 19920239] dictors of cardiovascular risk: the role of reclassification measures. Ann Intern 42. Hlatky MA. Framework for evaluating novel risk markers [Editorial]. Ann
Med. 2009;150:795-802. [PMID: 19487714] Intern Med. 2012;156:468-9. [PMID: 22431679] 23. Adabag AS, Therneau TM, Gersh BJ, Weston SA, Roger VL. Sudden death
43. Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using
after myocardial infarction. JAMA. 2008;300:2022-9. [PMID: 18984889] risk stratification tables. Ann Intern Med. 2008;149:751-60. [PMID: 19017593] 24. Auer R, Bauer DC, Marques-Vidal P, Butler J, Min LJ, Cornuz J, et al;
44. Janssens ACJW, Ioannidis JPA, van Duijn CM, Little J, Khoury MJ;
Health ABC Study. Association of major and minor ECG abnormalities with
GRIPS Group. Strengthening the reporting of genetic risk prediction studies:
coronary heart disease events. JAMA. 2012;307:1497-505. [PMID: 22496264] The GRIPS Statement. Ann Intern Med. 2011;154:421-5. [PMID: 21403077] 25. Breteler MMB. Mapping out biomarkers for Alzheimer disease [Editorial].
45. Kaptoge S, Di Angelantonio E, Lowe G, Pepys MB, Thompson SG, Col-
JAMA. 2011;305:304-5. [PMID: 21245188] lins R, et al; Emerging Risk Factors Collaboration. C-reactive protein concen-
26. Buckley DI, Fu R, Freeman M, Rogers K, Helfand M. C-reactive protein as
tration and risk of coronary heart disease, stroke, and mortality: an individual a risk factor for coronary heart disease: a systematic review and meta-analyses for participant meta-analysis. Lancet. 2010;375:132-40. [PMID: 20031199] the U.S. Preventive Services Task Force. Ann Intern Med. 2009;151:483-95.
46. Kaptoge S, Di Angelantonio E, Pennells L, Wood AM, White IR, Gao P,
et al; Emerging Risk Factors Collaboration. C-reactive protein, fibrinogen, and
27. Chou R, Arora B, Dana T, Fu R, Walker M, Humphrey L. Screening
cardiovascular disease prediction. N Engl J Med. 2012;367:1310-20. [PMID: asymptomatic adults with resting or exercise electrocardiography: a review of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. 2011; 47. Kathiresan S, Melander O, Anevski D, Guiducci C, Burtt NP, Roos C,
155:375-85. [PMID: 21930855] et al. Polymorphisms associated with cholesterol and risk of cardiovascular events.
28. Cook NR. Biomarkers for prediction of cardiovascular events [Letter].
N Engl J Med. 2008;358:1240-9. [PMID: 18354102] JAMA. 2009;302:2089. [PMID: 19920231] 48. Kavousi M, Elias-Smale S, Rutten JHW, Leening MJG, Vliegenthart R,
29. Cornelis MC, Qi L, Zhang C, Kraft P, Manson J, Cai T, et al. Joint effects
Verwoert GC, et al. Evaluation of newer risk markers for coronary heart disease
of common genetic variants on the risk for type 2 diabetes in U.S. men and risk classification: a cohort study. Ann Intern Med. 2012;156:438-44. [PMID: women of European ancestry. Ann Intern Med. 2009;150:541-50. [PMID: 49. Keller T, Zeller T, Ojeda F, Tzikas S, Lillpopp L, Sinning C, et al. Serial
30. de Boer IH, Levin G, Robinson-Cohen C, Biggs ML, Hoofnagle AN,
changes in highly sensitive troponin I assay and early diagnosis of myocardial Siscovick DS, et al. Serum 25-hydroxyvitamin D concentration and risk for
infarction. JAMA. 2011;306:2684-93. [PMID: 22203537] major clinical disease events in a community-based population of older adults: a 50. Kengne AP, Echouffo-Tcheugui JB, Sobngwi E. Coronary artery calcium
cohort study. Ann Intern Med. 2012;156:627-34. [PMID: 22547472] for guiding statin treatment [Letter]. Lancet. 2012;379:312. [PMID: 22284653] 31. de Lemos JA, Drazner MH, Omland T, Ayers CR, Khera A, Rohatgi A,
51. Khera AV, Cuchel M, de la Llera-Moya M, Rodrigues A, Burke MF, Jafri
et al. Association of troponin T detected with a highly sensitive assay and cardiac
K, et al. Cholesterol efflux capacity, high-density lipoprotein function, and ath-
structure and mortality risk in the general population. JAMA. 2010;304:2503- erosclerosis. N Engl J Med. 2011;364:127-35. [PMID: 21226578] 12. [PMID: 21139111] 52. Kim WR, Biggins SW, Kremers WK, Wiesner RH, Kamath PS, Benson
32. deFilippi CR, de Lemos JA, Christenson RH, Gottdiener JS, Kop WJ,
JT, et al. Hyponatremia and mortality among patients on the liver-transplant
Zhan M, et al. Association of serial measures of cardiac troponin T using a
waiting list. N Engl J Med. 2008;359:1018-26. [PMID: 18768945] sensitive assay with incident heart failure and cardiovascular mortality in older 53. Kivima¨ki M, Batty GD, Hamer M, Ferrie JE, Vahtera J, Virtanen M, et al.
adults. JAMA. 2010;304:2494-502. [PMID: 21078811] Using additional information on working hours to predict coronary heart disease: 33. den Ruijter HM, Peters SAE, Anderson TJ, Britton AR, Dekker JM,
a cohort study. Ann Intern Med. 2011;154:457-63. [PMID: 21464347] Eijkemans MJ, et al. Common carotid intima-media thickness measurements in
54. Koller MT, Leening MJG, Wolbers M, Steyerberg EW, Hunink MGM,
cardiovascular risk prediction: a meta-analysis. JAMA. 2012;308:796-803.
Schoop R, et al. Development and validation of a coronary risk prediction model
for older U.S. and European persons in the Cardiovascular Health Study and the 34. Devereaux PJ, Chan MTV, Alonso-Coello P, Walsh M, Berwanger O,
Rotterdam Study. Ann Intern Med. 2012;157:389-97. [PMID: 22986376] Villar JC, et al; Vascular Events In Noncardiac Surgery Patients Cohort Eval-
55. Lubitz SA, Yin X, Fontes JD, Magnani JW, Rienstra M, Pai M, et al.
uation (VISION) Study Investigators. Association between postoperative tro-
Association between familial atrial fibrillation and risk of new-onset atrial fibril- ponin levels and 30-day mortality among patients undergoing noncardiac sur- lation. JAMA. 2010;304:2263-9. [PMID: 21076174] gery. JAMA. 2012;307:2295-304. [PMID: 22706835] 56. Lyssenko V, Jonsson A, Almgren P, Pulizzi N, Isomaa B, Tuomi T, et al.
35. Di Angelantonio E, Gao P, Pennells L, Kaptoge S, Caslake M, Thompson
Clinical risk factors, DNA variants, and the development of type 2 diabetes. N A, et al; Emerging Risk Factors Collaboration. Lipid-related markers and car-
Engl J Med. 2008;359:2220-32. [PMID: 19020324] diovascular disease prediction. JAMA. 2012;307:2499-506. [PMID: 22797450] 57. Manolio TA. Genomewide association studies and assessment of the risk of
36. Eddy DM, Adler J, Patterson B, Lucas D, Smith KA, Morris M. Individ-
disease. N Engl J Med. 2010;363:166-76. [PMID: 20647212] ualized guidelines: the potential for increasing quality and reducing costs. Ann 58. Martı´nez ME, Thompson P, Messer K, Ashbeck EL, Lieberman DA, Baron
Intern Med. 2011;154:627-34. [PMID: 21536939] JA, et al. One-year risk for advanced colorectal neoplasia: U.S. versus U.K. risk-
37. Farooq V, van Klaveren D, Steyerberg EW, Meliga E, Vergouwe Y, Chieffo
stratification guidelines. Ann Intern Med. 2012;157:856-64. [PMID: 23247939] A, et al. Anatomical and clinical characteristics to guide decision making between
59. Matsushita K, Mahmoodi BK, Woodward M, Emberson JR, Jafar TH, Jee
coronary artery bypass surgery and percutaneous coronary intervention for indi- SH, et al; Chronic Kidney Disease Prognosis Consortium. Comparison of risk
vidual patients: development and validation of SYNTAX Score II. Lancet. 2013; prediction using the CKD-EPI equation and the MDRD study equation for 381:639-50. [PMID: 23439103] estimated glomerular filtration rate. JAMA. 2012;307:1941-51. [PMID: 38. Fonarow GC, Pan W, Saver JL, Smith EE, Reeves MJ, Broderick JP, et al.
Comparison of 30-day mortality models for profiling hospital performance in 60. McEvoy JW. Coronary artery calcium score and cardiovascular event predic-
acute ischemic stroke with vs without adjustment for stroke severity. JAMA.
tion [Letter]. JAMA. 2010;304:741-2. [PMID: 20716732] 2012;308:257-64. [PMID: 22797643] 61. Meigs JB, Shrader P, Sullivan LM, McAteer JB, Fox CS, Dupuis J, et al.
39. Gulati A, Jabbour A, Ismail TF, Guha K, Khwaja J, Raza S, et al. Associ-
Genotype score in addition to common risk factors for prediction of type 2 ation of fibrosis with mortality and sudden cardiac death in patients with non- diabetes. N Engl J Med. 2008;359:2208-19. [PMID: 19020323] 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 129
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Research and Reporting Methods Net Reclassification Improvement: Literature Review and Clinician's Guide 62. Melander O, Newton-Cheh C, Almgren P, Hedblad B, Berglund G, Eng-
82. Wacholder S, Hartge P, Prentice R, Garcia-Closas M, Feigelson HS, Diver
stro¨m G, et al. Novel and conventional biomarkers for prediction of incident
WR, et al. Performance of common genetic variants in breast-cancer risk models.
cardiovascular events in the community. JAMA. 2009;302:49-57. [PMID: N Engl J Med. 2010;362:986-93. [PMID: 20237344] 83. Wilson PWF. Challenges to improve coronary heart disease risk assessment
63. Melander O, Newton-Cheh C, Wang TJ. Biomarkers for prediction of
[Editorial]. JAMA. 2009;302:2369-70. [PMID: 19952326] cardiovascular events—reply. JAMA. 2009;302:2090.
84. Wormser D, Kaptoge S, Di Angelantonio E, Wood AM, Pennells L,
64. Omland T, de Lemos JA, Sabatine MS, Christophi CA, Rice MM, Jablon-
Thompson A, et al; Emerging Risk Factors Collaboration. Separate and com-
ski KA, et al; Prevention of Events with Angiotensin Converting Enzyme In-
bined associations of body-mass index and abdominal adiposity with cardiovas- hibition (PEACE) Trial Investigators. A sensitive cardiac troponin T assay in
cular disease: collaborative analysis of 58 prospective studies. Lancet. 2011;377: stable coronary artery disease. N Engl J Med. 2009;361:2538-47. [PMID: 1085-95. [PMID: 21397319] 85. Wormser D, Di Angelantonio E, Sattar N, Collins R, Thompson S,
65. Palomaki GE, Melillo S, Bradley LA. Association between 9p21 genomic
Danesh J, et al. Body-mass index, abdominal adiposity, and cardiovascular risk –
markers and heart disease: a meta-analysis. JAMA. 2010;303:648-56. [PMID: reply. Lancet. 2011;378:228.
86. Yeboah J, McClelland RL, Polonsky TS, Burke GL, Sibley CT, O'Leary D,
66. Paynter NP, Chasman DI, Buring JE, Shiffman D, Cook NR, Ridker PM.
et al. Comparison of novel risk markers for improvement in cardiovascular risk
Cardiovascular disease risk prediction with and without knowledge of genetic assessment in intermediate-risk individuals. JAMA. 2012;308:788-95. [PMID: variation at chromosome 9p21.3. Ann Intern Med. 2009;150:65-72. [PMID: 87. Zethelius B, Berglund L, Sundstro¨m J, Ingelsson E, Basu S, Larsson A,
67. Paynter NP, Chasman DI, Pare´ G, Buring JE, Cook NR, Miletich JP, et al.
et al. Use of multiple biomarkers to improve the prediction of death from car-
Association between a literature-based genetic risk score and cardiovascular events diovascular causes. N Engl J Med. 2008;358:2107-16. [PMID: 18480203] in women. JAMA. 2010;303:631-7. [PMID: 20159871] 88. Zoungas S, Patel A, Chalmers J, de Galan BE, Li Q, Billot L, et al;
68. Peralta CA, Shlipak MG, Judd S, Cushman M, McClellan W, Zakai NA,
ADVANCE Collaborative Group. Severe hypoglycemia and risks of vascular
et al. Detection of chronic kidney disease with creatinine, cystatin C, and urine
events and death. N Engl J Med. 2010;363:1410-8. [PMID: 20925543] albumin-to-creatinine ratio and association with progression to end-stage renal 89. Greenland P, Alpert JS, Beller GA, Benjamin EJ, Budoff MJ, Fayad ZA,
disease and mortality. JAMA. 2011;305:1545-52. [PMID: 21482744] et al; American College of Cardiology Foundation/American Heart Association
69. Pischon T, Boeing H, Hoffmann K, Bergmann M, Schulze MB, Overvad
Task Force on Practice Guidelines. 2010 ACCF/AHA guideline for assessment
K, et al. General and abdominal adiposity and risk of death in Europe. N Engl J
of cardiovascular risk in asymptomatic adults: a report of the American College of Med. 2008;359:2105-20. [PMID: 19005195] Cardiology Foundation/American Heart Association Task Force on Practice 70. Pletcher MJ, Tice JA, Pignone M. Modeling cardiovascular disease preven-
Guidelines. Circulation. 2010;122:e584-636. [PMID: 21098428]
90. Mosca L, Benjamin EJ, Berra K, Bezanson JL, Dolor RJ, Lloyd-Jones DM,
tion [Letter]. JAMA. 2010;303:835. [PMID: 20197528] et al. Effectiveness-based guidelines for the prevention of cardiovascular disease in
71. Polak JF, Pencina MJ, Pencina KM, O'Donnell CJ, Wolf PA, D'Agostino
women—2011 update: a guideline from the American Heart Association. Circu- RB Sr. Carotid-wall intima-media thickness and cardiovascular events. N Engl J
lation. 2011;123:1243-62. [PMID: 21325087] Med. 2011;365:213-21. [PMID: 21774709] 91. Cai T, Tian L, Lloyd-Jones DM. Comparing costs associated with risk
72. Polonsky TS, McClelland RL, Jorgensen NW, Bild DE, Burke GL, Guerci
stratification rules for t-year survival. Biostatistics. 2011;12:597-609. [PMID: AD, et al. Coronary artery calcium score and risk classification for coronary heart
disease prediction. JAMA. 2010;303:1610-6. [PMID: 20424251] 92. Leening MJG, Cook NR. Net reclassification improvement: a link between
73. Ripatti S, Tikkanen E, Orho-Melander M, Havulinna AS, Silander K,
statistics and clinical practice. Eur J Epidemiol. 2013;28:21-3. [PMID: Sharma A, et al. A multilocus genetic risk score for coronary heart disease: case-
control and prospective cohort analyses. Lancet. 2010;376:1393-400. [PMID: 93. Pepe MS. Problems with risk reclassification methods for evaluating predic-
tion models. Am J Epidemiol. 2011;173:1327-35. [PMID: 21555714] 74. Rosenberg S, Elashoff MR, Beineke P, Daniels SE, Wingrove JA, Tingley
94. Mihaescu R, van Zitteren M, van Hoek M, Sijbrands EJ, Uitterlinden AG,
WG, et al; PREDICT (Personalized Risk Evaluation and Diagnosis in the
Witteman JCM, et al. Improvement of risk prediction by genomic profiling:
Coronary Tree) Investigators. Multicenter validation of the diagnostic accuracy
reclassification measures versus the area under the receiver operating characteristic of a blood-based gene expression test for assessing obstructive coronary artery curve. Am J Epidemiol. 2010;172:353-61. [PMID: 20562194] disease in nondiabetic patients. Ann Intern Med. 2010;153:425-34. [PMID: 95. Mu¨hlenbruch K, Heraclides A, Steyerberg EW, Joost HG, Boeing H,
Schulze MB. Assessing improvement in disease prediction using net reclassifica-
75. Schelbert EB, Cao JJ, Sigurdsson S, Aspelund T, Kellman P, Aletras AH,
tion improvement: impact of risk cut-offs and number of risk categories. Eur J et al. Prevalence and prognosis of unrecognized myocardial infarction determined
Epidemiol. 2013;28:25-33. [PMID: 23179629] by cardiac magnetic resonance in older adults. JAMA. 2012;308:890-6. [PMID: 96. Pepe MS, Janes H. Commentary: Reporting standards are needed for eval-
uations of risk reclassification. Int J Epidemiol. 2011;40:1106-8. [PMID: 76. Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D'Agostino
RB Sr, et al. Development of a risk score for atrial fibrillation (Framingham
97. McGeechan K, Macaskill P, Irwig L, Liew G, Wong TY. Assessing new
Heart Study): a community-based cohort study. Lancet. 2009;373:739-45.
biomarkers and predictive models for use in clinical practice: a clinician's guide.
Arch Intern Med. 2008;168:2304-10. [PMID: 19029492] 77. Selvin E, Steffes MW, Zhu H, Matsushita K, Wagenknecht L, Pankow J,
98. Greenland S. The need for reorientation toward cost-effective prediction:
et al. Glycated hemoglobin, diabetes, and cardiovascular risk in nondiabetic
comments on ‘Evaluating the added predictive ability of a new marker: From area adults. N Engl J Med. 2010;362:800-11. [PMID: 20200384] under the ROC curve to reclassification and beyond' by M. J. Pencina et al., 78. Steyerberg EW, Pencina MJ. Reclassification calculations for persons with
Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:199-206.
incomplete follow-up [Letter]. Ann Intern Med. 2010;152:195-6. [PMID: 99. Cook NR, Paynter NP. Performance of reclassification statistics in compar-
79. Tammema¨gi MC, Katki HA, Hocking WG, Church TR, Caporaso N,
ing risk prediction models. Biom J. 2011;53:237-58. [PMID: 21294152] Kvale PA, et al. Selection criteria for lung-cancer screening. N Engl J Med.
100. National Cholesterol Education Program (NCEP) Expert Panel on De-
2013;368:728-36. [PMID: 23425165] tection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult
80. Tangri N, Stevens LA, Griffith J, Tighiouart H, Djurdjev O, Naimark D,
Treatment Panel III). Third Report of the National Cholesterol Education Pro-
et al. A predictive model for progression of chronic kidney disease to kidney
gram (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High failure. JAMA. 2011;305:1553-9. [PMID: 21482743] Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation.
81. Tzoulaki I, Liberopoulos G, Ioannidis JPA. Assessment of claims of im-
2002;106:3143-421. [PMID: 12485966] proved prediction beyond the Framingham risk score. JAMA. 2009;302:2345- 101. Cook NR, Paynter NP, Eaton CB, Manson JE, Martin LW, Robinson
52. [PMID: 19952321] JG, et al. Comparison of the Framingham and Reynolds Risk scores for global
130 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Net Reclassification Improvement: Literature Review and Clinician's Guide Research and Reporting Methods cardiovascular risk prediction in the multiethnic Women's Health Initiative. Cir- 115. Siontis GCM, Tzoulaki I, Siontis KC, Ioannidis JPA. Comparisons of
culation. 2012;125:1748-56, S1-11. [PMID: 22399535] established risk prediction models for cardiovascular disease: systematic review.
102. Pencina MJ, D'Agostino RB Sr, Larson MG, Massaro JM, Vasan RS.
BMJ. 2012;344:e3318. [PMID: 22628003] Predicting the 30-year risk of cardiovascular disease: the Framingham Heart 116. Steyerberg EW. Clinical Prediction Models: A Practical Approach to De-
Study. Circulation. 2009;119:3078-84. [PMID: 19506114] velopment, Validation, and Updating. New York: Springer; 2009.
103. Takahara M, Katakami N, Kaneto H, Shimomura I. Risk categorization
117. Kramer AA, Zimmerman JE. Assessing the calibration of mortality bench-
for calculating net reclassification improvement [Letter]. Eur J Epidemiol. 2013; marks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med.
28:607-9. [PMID: 23839540] 2007;35:2052-6. [PMID: 17568333] 104. Ganna A, Reilly M, de Faire U, Pedersen N, Magnusson P, Ingelsson E.
118. Cox DR. Two further applications of a model for binary regression. Bi-
ometrika. 1958;45:562-5.
Risk prediction measures for case-cohort and nested case-control designs: an ap- 119. Cook NR, Paynter NP. Comments on ‘Extensions of net reclassification
plication to cardiovascular disease. Am J Epidemiol. 2012;175:715-24. [PMID: improvement calculations to measure usefulness of new biomarkers' by M. J.
Pencina, R. B. D'Agostino, Sr. and E. W. Steyerberg [Letter]. Stat Med. 2012; 105. Pepe MS, Fan J, Seymour CW, Li C, Huang Y, Feng Z. Biases introduced
31:93-5. [PMID: 21344474] by choosing controls to match risk factors of cases in biomarker research. Clin 120. Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk mod-
Chem. 2012;58:1242-51. [PMID: 22730452] els: Use of benchmark values to disentangle a case-mix effect from incorrect 106. Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. 3rd ed. Phil-
coefficients. Am J Epidemiol. 2010;172:971-80. [PMID: 20807737] adelphia: Lippincott Williams & Wilkins; 2008.
121. Cook NR. Comments on ‘Evaluating the added predictive ability of a new
107. Kavousi M, Leening MJG, Witteman JCM. Markers for prediction of
marker: From area under the ROC curve to reclassification and beyond' by M. J.
cardiovascular disease risk [Letter]. JAMA. 2012;308:2561. [PMID: 23268505] Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008; 108. Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Com-
27:191-5. [PMID: 17671959] ments on ‘Integrated discrimination and net reclassification improvements— 122. Paynter NP, Cook NR. A bias-corrected net reclassification improvement
Practical advice'. Stat Med. 2008;27:207-12. [PMID: 17569110] for clinical subgroups. Med Decis Making. 2013;33:154-62. [PMID: 23042826] 109. Pepe MS, Feng Z, Gu JW. Comments on ‘Evaluating the added predictive
123. Vickers AJ, Elkin EB, Steyerberg E. Net reclassification improvement and
ability of a new marker: From area under the ROC curve to reclassification and decision theory [Letter]. Stat Med. 2009;28:525-6. [PMID: 17907248] beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).
124. Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Stey-
Stat Med. 2008;27:173-81. [PMID: 17671958] erberg EW. Evaluation of markers and risk prediction models: overview of rela-
110. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not
tionships between NRI and decision-analytic measures. Med Decis Making.
rely on integrated discrimination improvement and net reclassification index. Stat 2013;33:490-501. [PMID: 23313931] Med. 2013. [PMID: 23553436] 125. Localio AR, Goodman S. Beyond the usual prediction accuracy metrics:
111. Brindle P, Emberson J, Lampe F, Walker M, Whincup P, Fahey T, et al.
reporting results for clinical decision making [Editorial]. Ann Intern Med. 2012;157:294-5. [PMID: 22910942] Predictive accuracy of the Framingham coronary risk score in British men: pro- 126. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski
spective cohort study. BMJ. 2003;327:1267. [PMID: 14644971] N, et al. Assessing the performance of prediction models: a framework for
112. Hense HW, Schulte H, Lo¨wel H, Assmann G, Keil U. Framingham risk
traditional and novel measures. Epidemiology. 2010;21:128-38. [PMID: function overestimates risk of coronary heart disease in men and women from Germany—results from the MONICA Augsburg and the PROCAM cohorts.
127. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluat-
Eur Heart J. 2003;24:937-45. [PMID: 12714025] ing prediction models. Med Decis Making. 2006;26:565-74. [PMID: 113. Koller MT, Steyerberg EW, Wolbers M, Stijnen T, Bucher HC, Hunink
MGM, et al. Validity of the Framingham point scores in the elderly: results from
128. Vickers AJ, Cronin AM, Begg CB. One statistical test is sufficient for
the Rotterdam study. Am Heart J. 2007;154:87-93. [PMID: 17584559] assessing new predictive markers. BMC Med Res Methodol. 2011;11:13.
114. Merry AH, Boer JM, Schouten LJ, Ambergen T, Steyerberg EW, Feskens
EJM, et al. Risk prediction of incident coronary heart disease in The Nether-
129. Pepe MS, Kerr KF, Longton G, Wang Z. Testing for improve-
lands: re-estimation and improvement of the SCORE risk function. Eur J Prev ment in prediction model performance. Stat Med. 2013;32:1467-82. [PMID: Cardiol. 2012;19:840-8. [PMID: 21551214] 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 131
Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Annals of Internal Medicine
Current Author Addresses: Drs. Leening and Witteman: Department
Author Contributions: Conception and design: M.J.G. Leening, E.W.
of Epidemiology, Erasmus MC - University Medical Center Rotterdam, Dr. Molenwaterplein 50, 3015 GE Rotterdam, the Netherlands.
Analysis and interpretation of the data: M.J.G. Leening, M.M. Vedder, Ms. Vedder and Dr. Steyerberg: Department of Public Health, Erasmus E.W. Steyerberg.
MC - University Medical Center Rotterdam, Dr. Molenwaterplein 50, Drafting of the article: M.J.G. Leening.
3015 GE Rotterdam, the Netherlands.
Critical revision of the article for important intellectual content: M.J.G.
Dr. Pencina: Department of Biostatistics and Bioinformatics, Duke Leening, M.M. Vedder, J.C.M. Witteman, M.J. Pencina, E.W.
Clinical Research Institute, Duke University, 2400 Pratt Street, Dur- ham, NC 27715.
Final approval of the article: M.J.G. Leening, M.M. Vedder, J.C.M.
Witteman, M.J. Pencina, E.W. Steyerberg.
Statistical expertise: M.J.G. Leening, M.J. Pencina, E.W. Steyerberg.
Obtaining of funding: E.W. Steyerberg.
Administrative, technical, or logistic support: M.J.G. Leening.
Collection and assembly of data: M.J.G. Leening, M.M. Vedder.
130. McGeechan K, Liew G, Macaskill P, Irwig L, Klein R, Sharrett AR, et al.
Risk prediction of coronary heart disease based on retinal vascular caliber (from
the Atherosclerosis Risk In Communities [ARIC] Study). Am J Cardiol. 2008;
102:58-63. [PMID: 18572036]
Appendix Figure 1. Summary of evidence search and
selection.
Publications citing articles of interest
(n = 1479)
Cited reference 15: 1119
Cited reference 19: 39
Cited reference 20: 168
Cited reference 21: 18
Cited reference 22: 135
Duplicates excluded (n = 229)
Unique citations (n = 1250)
Selected citations (n = 66)
New England Journal of Medicine:
The Lancet: 7
Journal of the American Medical Association: 28
Annals of Internal Medicine:
Added through hand-search (n = 1)
Publications included in the review (n = 67)
The search was last updated on 23 April 2013.
21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Appendix Table 1. List and Main Characteristics of the 67 Articles
Study, Year (Reference)
Article Type
Outcome of Interest/Topic
Cutoffs Used for NRI
Adabag et al, 2008 (23) Sudden death after MI Auer et al, 2012 (24) ECG abnormalities 7.5% and 15% at 7.5 y Breteler, 2011 (25) Buckley et al, 2009 (26) Chou et al, 2011 (27) Resting or exercise ECG Cook and Ridker, 2009 (22) 5%, 10%, and 20% at 10 y† Cornelis et al, 2009 (29) Genetic risk score de Boer et al, 2012 (30) 25-Hydroxyvitamin D level Composite of hip fracture, MI, 50 nmol/L vs. season-specific at cancer, and death de Lemos et al, 2010 (31) Cardiac structure and death deFilippi et al, 2010 (32) Heart failure and CVD death 10% and 20% at 10 y den Ruijter et al, 2012 (33) 5%, 10%, and 20% at 10 y Devereaux et al, 2012 (34) Death after noncardiac 1%, 5%, and 10% at 30 d Di Angelantonio et al, 2012 (35) Cholesterol, apolipoprotein, and 10% and 20% at 10 y Eddy et al, 2011 (36) Hypertension guidelines Farooq et al, 2013 (37) Coronary revascularization strategies Fonarow et al, 2012 (38) Not specified at 30 d Gulati et al, 2013 (39) Myocardial fibrosis Death and major arrhythmia 5%, 10%, and 20% at 5 y (death); 15% at 5 y (majorarrhythmia) Helfand et al, 2009 (40) CAC score; leukocyte count; periodontal disease; ABI; cIMT;and CRP, Lp(a), homocysteine,and fasting glucose levels Hingorani and Psaty, 2009 (41) Hlatky, 2012 (42) Janes et al, 2008 (43) Risk stratification tables Janssens et al, 2011 (44) Kaptoge et al, 2010 (45) CHD, stroke, and death Kaptoge et al, 2012 (46) CRP and fibrinogen levels 10% and 20% at 10 y Kathiresan et al, 2008 (47) Genetic risk score 10% and 20% at 10 y Kavousi et al, 2012 (48) CKD; leukocyte count; CAC score; 10% and 20% at 10 y† cIMT; PAD; PWV; and vWFantigen, NT-proBNP, fibrinogen,CRP, homocysteine, and uric acidlevels Keller et al, 2011 (49) Serial changes in troponin I level Kengne et al, 2012 (50) CRP level and CAC score Khera et al, 2011 (51) Cholesterol efflux capacity Kim et al, 2008 (52) Kivimäki et al, 2011 (53) 5% and 10% at 10 y Koller et al, 2012 (54) BMI, CRP level, cIMT, ABI, and 10% and 20% at 10 y Lubitz et al, 2010 (55) Familial atrial fibrillation Atrial fibrillation 5% and 10% at 8 y Lyssenko et al, 2008 (56) Genetic polymorphisms 10% and 20% at an unspecified Manolio, 2010 (57) Genetic risk prediction Martinez et al, 2012 (58) U.K. and U.S. guidelines Advanced colorectal dysplasia Number, type, and size of Matsushita et al, 2012 (59) CKD-EPI and MDRD equations eGFR of 90, 60, 45, 30, and 15 mL/min per 1.73 m2 at anunspecified horizon McEvoy, 2010 (60) Meigs et al, 2008 (61) Genetic risk score Melander et al, 2009 (62) CRP, cystatin C, Lp-PLA2, 6%, 10%, and 20% at 10 y MR-proADM, MR-proANP, andNT-proBNP levels Melander et al, 2009 (63) Omland et al, 2009 (64) CVD death, heart failure, and Palomaki et al, 2010 (65) Chromosome 9p21 polymorphisms 5%, 10%, and 20% at 10 y† Paynter et al, 2009 (66)‡ Chromosome 9p21.3 polymorphisms 5%, 10%, and 20% at 10 y† Paynter et al, 2010 (67) Genetic risk score 5%, 10%, and 20% at 10 y Peralta et al, 2011 (68) Creatinine level, cystatin C level, and Continuous NRI at an unspecified urine albumin–creatinine ratio Continued on following page 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Study, Year (Reference)
Article Type
Outcome of Interest/Topic
Cutoffs Used for NRI
Pischon et al, 2008 (69) BMI and abdominal adiposity 2.5%, 5%, and 7.5% at 5 y Pletcher et al, 2010 (70) Polak et al, 2011 (71) 6% and 20% at 10 y† Polonsky et al, 2010 (72) 3% and 10% at 5 y Ripatti et al, 2010 (73) Genetic risk score 5%, 10%, and 20% at 10 y Rosenberg et al, 2010 (74) Gene expression test Presence of obstructive CAD Schelbert et al, 2012 (75) Continuous NRI at an unspecified Schnabel et al, 2009 (76) Atrial fibrillation 5% and 15% at 10 y Selvin et al, 2010 (77) Glycated hemoglobin level Type 2 DM, CHD, and death 5%, 10%, and 20% at 10 y Steyerberg and Pencina, 2010 (78) 5%, 10%, and 20% at 10 y† Tammemägi et al, 2013 (79) Smoking intensity and history of Tangri et al, 2011 (80) Calcium, phosphate, bicarbonate, and albumin levels Tzoulaki et al, 2009 (81) Wacholder et al, 2010 (82) Genetic polymorphisms Wilson, 2009 (83) Wormser et al, 2011 (84) BMI and abdominal adiposity 5%, 10%, and 20% at 10 y Wormser et al, 2011 (85) BMI and abdominal adiposity Yeboah et al, 2012 (86) cIMT, CAC score, brachial FMD, 5% and 20% at 10 y† ABI, CRP level, and family history Zethelius et al, 2008 (87) Troponin I, NT-proBNP, cystatin C, 6% and 20% at an unspecified Zoungas et al, 2010 (88) Severe hypoglycemia ABI ⫽ ankle– brachial index; BMI ⫽ body mass index; CAC ⫽ coronary artery calcium; CAD ⫽ coronary artery disease; CHD ⫽ coronary heart disease; cIMT ⫽ carotid intima–media thickness; CKD ⫽ chronic kidney disease; CKD-EPI ⫽ Chronic Kidney Disease Epidemiology Collaboration; CRP ⫽ C-reactive protein; CVD ⫽ cardiovascular disease; DM ⫽ diabetes mellitus; ECG ⫽ electrocardiography; ECG-LVH ⫽ electrocardiographic left ventricular hypertrophy; eGFR ⫽ estimated glomerular filtration rate; ESLD ⫽ end-stage liver disease; ESRD ⫽ end-stage renal disease; FMD ⫽ flow-mediated dilation; GRIPS ⫽ Genetic Risk Prediction Studies; Lp(a) ⫽ lipoprotein(a); Lp-PLA2 ⫽ lipoprotein-associated phospholipase A2; MDRD ⫽ Modification of Diet in Renal Disease; MI ⫽ myocardial infarction; MR-proADM ⫽ midregional proadrenomedullin; MR-proANP ⫽ midregional proatrial natriuretic peptide; NA ⫽ not applicable; NIH ⫽ National Institutes of Health; NRI ⫽ net reclassification improvement; NT-proBNP ⫽ N-terminal fragment of prohormone B-type natriuretic peptide; PAD ⫽ peripheral arterial disease; PWV ⫽ pulse wave velocity; vWF ⫽ von Willebrand factor.
* NRI was not calculated.
† Observations from a follow-up period shorter than the predicted time horizon were used.
‡ Identified through hand-search with erroneous citation linkage to a methodological article on NRI.
21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Appendix Table 2. Summary Characteristics of the 67
Articles
Studies, n (%)
New England Journal of Medicine The Lancet Journal of the American Medical Association Annals of Internal Medicine Year of print publication
Cited methodological article
Pencina et al, 2008 (15) Pencina et al, 2010 (19) Pencina et al, 2011 (20) Pencina et al, 2012 (21) Cook and Ridker, 2009 (22) Country of address for correspondence
Appendix Figure 2. Example of a reclassification graph with superimposed cut points of predicted risk.
ariables an
10-y Risk Pre
taining Framingham V
CHD event
10-y Risk Predicted by Model Containing Only Framingham Variables
The graph shows 10-y risk for incident CHD in women from the ARIC (Atherosclerosis Risk in Communities) Study predicted by a model containing
only the Framingham risk score variables (horizontal axis) against risk predicted by a model containing Framingham risk score variables and retinal
arteriolar caliber (vertical axis). Lines at predicted risks of 10% and 20% are superimposed to show reclassification over clinically relevant cut points (2,
89) and thereby create a visual representation of a reclassification table (Appendix Table 3). Of note, most women in this study have a low (⬍10%)
predicted risk for CHD, both with the Framingham variables and with the model that includes retinal arteriolar caliber. The graph also shows that alimited number of women are reclassified over the cut points (i.e., only a small proportion of dots lies in the off-diagonal cells of the graph). CHD ⫽ coronary heart disease. (Reproduced from McGeechan and colleagues [130] with permission of the American Journal of Cardiology.) 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014
Appendix Table 3. Example of a Risk Reclassification Table Stratified by Event Status*
Model Containing Only Framingham
Model Containing Framingham Risk Score Variables and Coronary Artery Calcium Score
Risk Score Variables
<10% Risk
>20% Risk
<10% risk
Persons with event, n Persons without event, n Total persons, n Observed risk (95% CI), % Persons with event, n Persons without event, n Total persons, n Observed risk (95% CI), % >20% risk
Persons with event, n Persons without event, n Total persons, n Observed risk (95% CI), % Total persons, n
* The Table shows reclassification for 10-y risk for incident coronary heart disease in participants from the Rotterdam Study predicted by a model containing only theFramingham risk score variables against risk predicted by a model containing Framingham risk score variables and coronary artery calcium score. The numbers are roundeddue to the use of Kaplan–Meier estimates for persons with incomplete follow-up. (Reproduced from reference 48.) 21 January 2014 Annals of Internal Medicine Volume 160 • Number 2 Downloaded From: http://annals.org/ by a Erasmus MC User on 03/14/2014

Source: https://survey.erasmusmc.nl/wiki/mgz-clinpred/lib/exe/fetch.php?id=markers%3Astart&cache=cache&media=markers:leening_nri_review_annintmed2014.pdf