How To Determine Minimum Sample Size In R

AB Sample Size Calculation in R

Some useful tools in R for calculating sample size and experimentation elapsing

Introduction

Performing advert-hoc analysis for stakeholders can be fourth dimension consuming. Furthermore, there are a few questions that I go asked on a reasonably frequent footing. So I accept been spending some fourth dimension developing some tools for my "non-technical" colleagues to utilise in R.

One of the most unremarkably asked questions is "How big of a sample do I need to achieve significance?", which is oftentimes followed past "How long do I need to run my experiment for?". For this reason, I have developed some unproblematic code for people to use when they demand to answer these questions. All the user needs to do is pass some baseline numbers into some functions I take created and they tin can decide their sample size requirements and experiment duration on an advertizement-hoc basis.

Sample size, statistical power and experiment elapsing

Luckily, by knowing a few simple pieces of information the pwr() package in R can answer these two questions with a fair amount of ease. Pwr() helps you perform power analysis prior to conducting an experiment, which enables you to determine how big your sample size should be per experimental status.

The four quantities required to compute power analysis accept an intimate human relationship and we are able to compute whatsoever one of these values if nosotros have the remaining inputs:

one. sample size (n)

2. effect size

3. significance level (alpha)= P(Type I error) = probability of finding an effect that is not there

4. power = 1 — P(Type 2 error) = probability of finding an effect that is there

As your significance level (iii) and ability (4) are typically fixed values, as long every bit you can input the effects sizes (2) for your control and variant, you tin can determine your required sample size (1).

Thankfully, the ES.h() function in the pwr() bundle computes our effect size for united states to pass into power analyses. We will typically know the current conversion rate/performance of our control condition but the effect of the variant is about by definition an unknown. All the same, nosotros can summate an expected result size, given a desired uplift. Once these effects are computed they are passed into the pwr.p.test() function which will compute our sample size, providing n is left blank. To make this sort of analysis user friendly, I take wrapped both aforementioned functions into a new function chosen sample_size_calculator().

Furthermore, as we will use this information to then calculate the number of days needed to run the experiment, I have created a days_calculator() function also, which will use the output from our sample size adding:

          sample_size_calculator <- function(control, uplift){
variant <- (uplift + 1) * control
baseline <- ES.h(control, variant)
sample_size_output <- pwr.p.exam(h = baseline,
n = ,
sig.level = 0.05,
power = 0.8)
if(variant >= 0)
{render(sample_size_output)}
else
{paste("N/A")}
}          days_calculator <- function(sample_size_output, average_daily_traffic){
days_required <- c(sample_size_output * 2)/(average_daily_traffic)
if(days_required >= 0)
{paste("It will take this many days to attain significance with your current traffic:", round(days_required, digits = 0))}
else
{paste("N/A")}
}

If yous are using this tool, yous just specify your command conversion rate and desired uplift:

          command <- 0.034567          uplift <- 0.01

And run the sample_size_calculator() role:

          sample_size_calculator(control, uplift)          sample_size_output <- sample_size_output$northward          sample_size_output

You will then get your required sample size output given these values (retrieve this sample size requirement is per variant):

          [n]230345

Now nosotros have this information we can determine how long the experiment needs to run for. All that yous will need to input is your average daily traffic:

          average_daily_traffic <- 42000

Run the days_calculator() part:

          days_calculator(sample_size_output, average_daily_traffic)

And you volition become the following output:

          [one] It will have this many days to attain significance with your current traffic: 36

Although this code is just relevant if you are conducting an experiment with an AB design (i.e with only two experimental conditions), the functions presented tin can be amended to calculate the required sample size given multiple experimental conditions, using the pwr.anova.exam() function within sample_size_calculator(), replacing pwr.2p.test().

Determination

Ability analysis is an imperative aspect of any experimental blueprint. Information technology allows analysts to decide the required sample size needed to notice a statistically significant effect of a given size, with a given degree of confidence. Conversely, information technology likewise facilitates the detection an effect of a given size with a given level of confidence, under sample-size constraints. If the probability is depression, it could be advisable to alter the experimental design of your experiment or to minimise certain numerical values that are input into your power analyses.

Used in conjunction with ane some other, calculating both your required sample and experimentation duration can exist incredibly useful information to provide to stakeholders. Obtaining this information tin can assist them efficiently plan their experimentation road-maps. Furthermore, these predetermined numbers tin can assist in determining the feasibility of certain experiments or whether the uplifts desired are too idealistic.