predint.md

Prediction intervals with missing data

authors: Florian van Leeuwen, Thom Benjamin Volker, Gerko Vink and Stef van Buuren

The development and application of (clinical) prediction models is complicated by missing data, as most analysis techniques do not readily allow for incorporating missing values. Consequently, model parameters cannot be estimated, and predictions cannot be calculated. Ad-hoc fixes to deal with missing data, such as listwise deletion or mean imputation, work only under limited circumstances, such as MCAR, which are unlikely to hold in practice. A more principled approach dealing with missing data is multiple imputation (MI). Many studies confirmed that MI allows one to obtain unbiased and efficient estimates of model parameters under fairly general conditions. Practitioners in (clinical) prediction commonly conceive single imputation to be sufficient. The present study compares single versus multiple imputation for making point estimates (predictions) and prediction intervals that quantify the uncertainty of that prediction.

We calculated predictions and prediction intervals for single imputation by standard methods. For multiple imputation, we applied Rubin’s rules to pool predictions and developed a method for estimating prediction intervals for linear regression problems, and checked whether intervals for cases with more missing values were wider. We investigated the bias of the predictions and the coverage of the predictions intervals by simulation.

Multiple imputation outperformed the single method. We found that multiple imputation yields confidence-valid prediction intervals, whereas interval created by single imputation severely underestimated the uncertainty around a prediction, an effect that occurred regardless missing values were located in the training and/or testing phase, although the problem is more severe in case of the former (see the figure below).

Hence, single imputation for prediction yields poor estimates of uncertainty under missing data. On the other hand, multiple imputation produces correct prediction intervals. We recommend the use of multiple imputation for prediction tasks that require an estimate of uncertainty around the estimates.