Powering Up Cognitive Psychology: Comparing Statistical Methods for Detecting the SNARC Effect via Virtual Experiments

Generalized linear mixed-effects models (GLMMs) are increasingly used to analyze reaction time (RT) data in cognitive psychology. Yet, whether they genuinely outperform linear mixed-effects models (LMMs) or t-tests on aggregated data in realistic experimental settings remains unclear. In this preregistered simulation study, we systematically compared three types of GLMMs – a Gamma GLMM, an Inverse Gaussian GLMM, and a novel Probit GLMM – to LMMs and the t-test on aggregated data. As the main performance metric, we introduced adjusted power, a measure that adjusts statistical power to equalize methods in terms of confidence interval coverage. To this end, we used virtual experiments derived from five large datasets: three from numerical cognition experiments investigating the SNARC effect (Spatial-Numerical Association of Response Codes), and two from inhibitory control tasks. Across 1,480,000 simulations covering 36 configurations and including RTs from over 2,000 participants, we found that Gamma and Probit GLMMs outperform LMMs and the t-test on average, with the Probit GLMM showing a particular advantage for detecting the SNARC effect. However, no single method dominates across all configurations: performance depends substantially on the dataset and effect considered, and some mixed-effects models require robust error estimation to provide adequate coverage at small sample sizes. To make these findings actionable, we provide the interactive Shiny app Powering Up Cognitive Psychology, which allows researchers to explore how the results depend on analytical choices and to make informed decisions about which statistical method to use for their data analysis given their own experimental context.