Suppose I have a two phase experiment. The goal of the experiment will be to test if there are differences in proportions between two treatments. In phase one, I have no idea how many samples I will need as I have no prior information, so say I take 30 samples for each treatment and get proportions of 0.5 and 0.6. Now I want to use this information to calculate the number of samples I will need in phase two in order to have an 80% chance (power) to show a difference between the two treatments (assume alpha = 0.05). The sample size calculators I have found online do not apply as you to enter in the population proportions, but I don't have the population proportions, only estimated proportions based on the first sample. So I need to take the sample variation into account somehow in the sample size calculation. Any help would be greatly appreciated.
In various statistical software programs (and, allegedly, in some online 'calculators') you can specify typical proportions that you'd like to be able to distinguish at the 5% level of significance and with power 80%.
Specifically, if reasonable proportions for Treatments 1 and 2 are $p_1 = 0.5$ and $p_2 = 0.6,$ then these are the 'proportions' you enter. (Of course, you won't know the exact proportions, but the difference between them should be the size of difference you'd like to be able to detect.)
Sample size computation from Minitab. In particular, output from a 'power and sample size' procedure in a recent release of Minitab is shown below. For a two-sided test with the proportions guessed above, you'd need $n=388$ in each group for 80% power.
Power and Sample Size Test for Two Proportions Testing comparison p = baseline p (versus ≠) Calculating power for baseline p = 0.5 α = 0.05 Sample Target Comparison p Size Power Actual Power 0.6 388 0.8 0.800672 The sample size is for each group.
Often tests to distinguish between two binomial proportions are done in terms of approximate normal tests, which are quite accurate for sample sizes this large and for success probabilities not too near to $0$ or $1.$
Example of test of two proportions. Suppose that your results are $183$ in the first group and $241$ in the second. Then Minitab's version of the one-sided test shows a highly significant difference with a P-value near $0.$
Test and CI for Two Proportions Sample X N Sample p 1 182 388 0.469072 2 241 388 0.621134 Difference = p (1) - p (2) Estimate for difference: -0.152062 95% CI for difference: (-0.221312, -0.0828117) Test for difference = 0 (vs ≠ 0): Z = -4.30 P-Value = 0.000 Similar test in R: For comparison, the version of the test implemented in the R procedure 'prop.test' gives the following result, also leading to rejection of the null hypothesis. (I use the version without continuity correction on account of the large sample size.)
prop.test(c(182,241), c(388,388), cor=F) 2-sample test for equality of proportions without continuity correction data: c(182, 241) out of c(388, 388) X-squared = 18.091, df = 1, p-value = 2.106e-05 alternative hypothesis: two.sided 95 percent confidence interval: -0.22131203 -0.08281168 sample estimates: prop 1 prop 2 0.4690722 0.6211340 Simulation of power. The following simulation in R with 'prop.test' shows that the power of the test to distinguish between proportions $0.5$ and $0.6$ at the 5% level is roughly 80%.
set.seed(112) pv = replicate(10^5, prop.test(rbinom(2,388,c(.5,.6)),c(388,388),cor=F)$p.val) mean(pv 