Jacob Coutts
Department of Psychology
The Ohio State University
Title: BS Power: Bootstrap or …?
Abstract: Power analysis is used as the gold standard for justifying sample sizes in the design and pre-registration of studies. Effect sizes used in power analyses are often informed by estimates taken from previous studies, but these estimates come with sampling variability. The classical approach to power analysis ignores this uncertainty and, consequently, the sample size outputs from a power calculation are often wrong. New-generation power analysis methods attempt to incorporate the uncertainties that limit the usefulness of classical power analysis, but these modern methods are either highly conservative or specified for only a limited number of simple procedures. The present talk evaluates the nonparametric bootstrap and Monte Carlo resampling procedures as a flexible alternative to incorporating effect size variability in power calculations. The resampling methods were evaluated by how well they capture important target values and how well they perform relative to the classical and three new-generation approaches to power analysis: safeguard power (Perugini, Gallucci, & Costantini, 2014), power calibrated effect size (PCES; McShane & Böckenholt, 2016), and bias and uncertainty corrected sample size (BUCSS; Taylor & Muller, 1996; Anderson, Kelley, & Maxwell, 2017). Preliminary results suggest that: 1.) less confidence should be placed on power analyses in general due to the uncertainty of effect size estimates, 2.) resampling procedures may provide an alternative to power/sample size calculations, and 3.) there is room for improvement in the pedagogy of power analysis. Future directions for the project are discussed.
Discussant: Ivory Li