A Comprehensive Review of Missing Data Handling: From Traditional Methods to Modern AI-driven Solutions
Abstract
This is the problem of incomplete datasets, which is highly common and persistent in all spheres of empirical inquiry and casts a fundamental challenge on the quality of statistical inferences and predictive capabilities of machine learning models. This review gives a detailed analysis of the development of techniques that are applied to manage missing data, their primitive methods of historical analysis to modern and sophisticated techniques. Our discussion begins with a description of the necessary typology of missing data mechanisms, Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR), as it offers the theory behind selecting an appropriate imputation strategy. The paper subsequently follows up the history of the traditional techniques of statistics, showing the inadequacy of such simplistic approaches to the data as case deletion and single imputation in terms of the resilient, uncertainty-based Multiple Imputation (MI), which continues to be considered the standard of MAR data. Next, we discuss the new paradigm of AI and ML-based models, including Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformers that have shown themselves to be highly effective at representing complex non-linear patterns in complex high-dimensional data. There is a particular emphasis on practical challenges of handling the heterogeneous types of data. The analysis we do in comparison reveals that no universal solution is universally best, and the choice will have to be informed by the type of the missingness, as well as the arrangement of the data, and the analytic goals. The review ends with the statement that even though modern AI models can make spectacular predictions, much work is required to establish reliable approaches to MNAR settings, formalize uncertainty estimation in deep learning imputations, and create standardized evaluation methods of methods.
Keywords: Missing Data, Imputation, Multiple Imputation, Machine Learning, Deep Learning