In traditional statistical methodology (e.g., analysis of variance), confidence in the observed results is often assessed by computing power. In most cases, adding more participants to a study will improve power more than increasing the amount of data collected from each participant. Thus, traditional statistical methods are biased in favor of experiments with large numbers of participants. Here we propose a method for computing confidence in the results of experiments in which much data is collected from few participants. In such experiments it is common to fit a series of mathematical models to the resulting data and to conclude that the best fitting model is superior. The probability of replicating this conclusion (i.e., Prep) is derived for any two nested models. Simulations and empirical applications of this new statistic confirm its utility as an alternative to power analyses in studies where much data is collected from few participants. |