A Comparative Simulation Study of Classical and Bayesian Regression Models for Complex Non-Linear Data Generation Processes

Abstract

This paper presents a comprehensive simulation study comparing the performance of classical and Bayesian regression methods when applied to data generated from complex, non-linear processes with heteroscedastic errors. Using extensive Monte Carlo simulations, we evaluate classical linear regression, Bayesian linear regression, Bayesian regularized regression with Horseshoe priors, Bayesian Generalized Additive Models (GAMs), and Bayesian model averaging. Our findings indicate that Bayesian methods, particularly flexible approaches like Bayesian GAMs, offer superior uncertainty quantification, better-calibrated prediction intervals, and greater robustness to model misspecification than classical approaches. The Bayesian framework naturally accommodates complex data structures and provides principled uncertainty estimates, making it highly valuable for applied research where understanding the full distribution of potential outcomes is critical.