Dealing with Distributional Assumptions in Preregistered Research

Abstract

Virtually any inferential statistical analysis relies on distributional assumptions of some kind. The violation of distributional assumptions can result in consequences ranging from small changes to error rates through to substantially biased estimates. Conventionally, researchers have conducted assumption checks after collecting data, and then changed the primary analysis technique if distributional problems are observed. An approach to dealing with distributional assumptions that requires decisions to be made contingent on observed data is problematic, however, in preregistered research, where researchers attempt to specify all important analysis decisions prior to collecting data. Limited methodological advice is currently available about how to deal with the prospect of distributional assumption violations in preregistered research. In this article, we examine several strategies that researchers could use in preregistrations to reduce the potential impact of distributional assumption violations. We suggest that pre-emptively selecting analysis methods that are as robust as possible to assumption violations, performing planned sensitivity analyses, and/or supplementing preregistered confirmatory analyses with exploratory checks of distributional assumptions may all be useful strategies. On the other hand, we suggest prespecifying ‘decision trees’ for selecting data analysis methods based on the distributional characteristics of the data may not be practical in most situations.