How To Seize pwr: Statistical Power Analysis in R

Statistical Power analysis is a critical part of designing a study or experiment. It lets you balance the cost of an experiment with the anticipated value of the results. The R language has a module, pwr, which you can use to model these trade-offs in a simulated data model called a power simulation.

To introduce the topic, real world experiments are a balancing act. Every experiment involves selecting a combination of the following three factors.

Cost – often driven by required sample size
Ability to detect small effects
Confidence that the outcome reflects the underlying process

These three factors give rise to a fourth: power.

The statistical power of an experiment represents the probability of identifying an interaction effect on the dependent variable that is present in the population you are sampling.

Power Analysis – The pwr package

We can use the pwr package to perform statistical power analysis in R.

This package has statistical power analyses for many experiment or study types. These have a common approach: enter three of the four parameter options above (sample size, effect size, statistical significance, and power) and the package will calculate the fourth parameter.

The package has some defaults. The significance level is defaulted to a=0.05.

There is no simple answer to the question selecting a desired true effect size. For those of you working in industry, there is an obvious solution: consult with your financial team and executive sponsors to identify what level of result is a financially meaningful factor, or a statistically significant result. For example, a 2% reduction in manufacturing waste could be perceived as a significant contribution to the organization. Whereas we might need a 15% to 20% lift in customer satisfaction to justify adding new features to a product.

A statistical test like power analyses can help you make multiple comparisons across simulated data sets, finding the true effect of the hypothesis on the dependent variable and other statistical test measures. Even across unequal sample sizes, you can measure the mean, standard deviation, and confidence interval of the desired interaction effect, and perform a hypothesis test against the null hypothesis as if all of the simulated data had an equal sample size.

Statistical Power Analysis Options

R’s pwr package supports the following:

function	Supports	Implementation
pwr.f2.test	General Linear model	pwr.f2.test(u =, v = , f2 = , sig.level = , power = )
pwr.r.test	Correlation Analysis	pwr.r.test(n = , r = , sig.level = , power = )
pwr.anova.test	One Way ANOVA	pwr.anova.test(k = , n = , f = , sig.level = , power = )
pwr.chisq.test	chi-square test	pwr.chisq.test(w =, N = , df = , sig.level =, power = )
pwr.t.test	one sample t-test	pwr.t.test(n = , d = , sig.level = , power = , type = c(“two.sample”, “one.sample”, “paired”))
pwr.t2n.test	two sample t-test	pwr.t2n.test(n1 = , n2= , d = , sig.level =, power = )
pwr.p.test	proportions (one sample)	pwr.p.test(h = , n = , sig.level = power = )
pwr.2p.test	proportions (two samples, equal n)	pwr.2p.test(h = , n = , sig.level =, power = )
pwr.2p2n.test	proportions (two samples, different n)	pwr.2p2n.test(h = , n1 = , n2 = , sig.level = , power = )

Pwr Examples

# power analysis in r example
library(pwr)

# Election Example. Interpreting a poll result.
# Assume a significance level of .05 and a sample of
# 1000 voters. What effect size can be detected with
# a power of .5?

> pwr.2p.test(n=1000,sig.level=0.05,power=0.5)

     Difference of proportion power calculation test statistic for binomial distribution (arcsine transformation) 

              h = 0.08764393
              n = 1000
      sig.level = 0.05
          power = 0.5
    alternative = two.sided

NOTE: equal sample size

For a second example, let us assume we’re trying to measure public response to an issue; a single sample test statistic of proportions would be appropriate here.

# power analysis in r example
> pwr.p.test(n=1000,sig.level=0.05,power=0.5)

     proportion power calculation for binomial distribution (arcsine transformation) 

              h = 0.06196988
              n = 1000
      sig.level = 0.05
          power = 0.5
    alternative = two.sided

Which can be improved upon by the simple act of boosting the required sample size.

# power analysis in r example
> pwr.p.test(n=5000,sig.level=0.05,power=0.5)

     proportion power calculation for binomial distribution (arcsine transformation) 

              h = 0.02771587
              n = 5000
      sig.level = 0.05
          power = 0.5
    alternative = two.sided

Related Materials