Table 3

DemPoRT (Dementia Population Risk Tool) goodness of fit summary statistics of the full model in the derivation, validation and combined data, and the reduced model in the combined data*

DerivationValidationCombinedReduced
Male model
Discrimination
 C-statistic (95% CI)0.82 (0.80 to 0.83)0.83 (0.81 to 0.85)0.82 (0.81 to 0.84)0.82 (0.81 to 0.84)
 Ratio of 95 to 5 risk percentile28.6 (0.128/0.004)26.1 (0.119/0.005)36.5 (0.134/0.004)36.2 (0.134/0.004)
Calibration
 Observed vs predicted1 −0.76%4.21%−0.61%−0.61%
 5-year cumulative incidence
 (observed) (95% CI)
0.044 (0.041 to 0.047)0.045 (0.041 to 0.048)0.044 (0.042 0.047)0.044 (0.042 to 0.047)
 5-year risk (predicted)0.0450.0430.0450.045
 Calibration slope and intercept0.7859 to 0.00980.7799 to 0.00800.8240 to 0.00810.8285 to 0.0079
Overall performance
 Brier Score (scaled)0.0810.0680.0860.086
 Nagelkerke R2 0.1050.1230.1000.100
Female model
Discrimination
 C-statistic (95% CI)0.82 (0.81 to 0.83)0.83 (0.81 to 0.85)0.83 (0.82 to 0.83)0.83 (0.82 to 0.83)
 Ratio of 95 to 5 risk percentile54.3 (0.171/0.003)50.6 (0.167/0.003)64.8 (0.178/0.003)64.6 (0.178/0.003)
Calibration
 Observed vs predicted*−1.07%−10.58%−0.78%−0.78%
 5-year cumulative incidence
 (observed) (95% CI)
0.060 (0.057 to 0.062)0.053 (0.050 to 0.057)0.057 (0.055 to 0.060)0.057 (0.055 to 0.060)
 5-year risk (predicted)0.0600.0590.0580.058
 Calibration slope and intercept0.7671 to 0.01450.8666 to 0.01280.8320 to 0.01020.8335 to 0.0101
Overall performance
 Brier Score (scaled)0.1070.1020.1110.111
 Nagelkerke R2 0.1470.1320.1330.133
  • *Three types of performance tests were examined.22 (1) Discrimination is the ability of a predictive model to differentiate between those who experience the outcome from those who do not. C-statistic is a rank order statistic for predictions against true outcomes.24 The statistic ranges from 0 to 1; a value of 0.5 indicates the model is no better than random prediction, while a value of 1 indicates the model perfectly predicts whose who will develop the outcome of interest and who will not. Ratio of 95 to 5 risk percentile is a measure indicating the spread of the predicted risks, where a higher ratio indicates a more discriminating algorithm. For example, a ratio of 20 indicates that the absolute risk of the event of interest is 20 times higher for a person in the 95th percentile of risk than for a person in the 5th percentile of risk. (2) Calibration (or accuracy) describes how well the predicted probability of disease agrees with the observed outcomes. Observed versus predicted (O vs P) is the relative difference between the observed incidence and the predicted risk, calculated as (Observed – Predicted)/Observed × 100. The absolute values for this calculation are the observed 5-year cumulative incidence and the predicted 5-year risk. A 1% difference indicates that 1% more events were observed than were predicted. This tables show overall O vs P. Online supplemental digital content 6 and 7 show O vs P for specific subgroups. The calibration slope and intercept indicates the slope and intercept of the calibration plots. Figure 1 displays the calibration plots for validation data. (3) Overall performance measures. Brier score (scaled) is a measure of overall agreement between observed and predicted risk with values between 0 and 1.33 Nagelkerke R2 is a measure of the amount of variation in risk between individuals in the data that is explained by the model, with values from 0 to 132. Larger values indicate that more variation is explained.