The Twitter engineering blog post explaining the fallacy of trying to use two control buckets to limit false positives includes the following probability distribution tables of various statistical significance outcomes for an experiment containing three buckets, A1 (control), A2 (control), and B (treatment), where B is drawn from the same underlying distribution as both controls -- the null hypothesis is true.

In this appendix, we provide the analytical derivation of these probabilities.

Numbers in the purple table are derived using Proposition 2, which we prove below. Here, X, Y, and Z refer to our three buckets -- control A1, control A2, and treatment B.

Proposition 2 provides the asymptotic joint distribution of the three t statistics when

1. X, Y and Z are sampled with equal probability from the total population, for example, 5%, 5% and 5%.
2. X, Y and Z share the same distribution mean and distribution variance.

From there we numerically calculate the conditional distributions under the cases X=Y, X>Y and X<Y respectively (as is shown in the purple table). For other cases, the table can be derived using Proposition 1.

Lemma 1 : If X_n converges in distribution to a constant C, then X_n converges in probability to C.

Lemma 2 (Slutsky Theorem) : Let {X_n}, {Y_n} be sequences of scalar/vector/matrix random variables.

If X_n converges in distribution to a random element X and Y_n converges in probability to a nonzero constant C, then X_n/Y_n converges in distribution to X/C.

Proposition 1: X,Y,Z are independent distributions sharing the same mean and they have standard deviations 0 <σ ₁, σ₂, σ₃ < ∞. Draw n₁, n₂, n₃ samples from each of them independently, and let n=n₁+n₂+n₃. Denote s₁, s₂, s₃ to be the sample variance of X,Y,Z and X‾, Y‾, Z‾ to be the sample mean. If there exists R₁, R₂ and R₃ such that

we have the following weak convergence:

Proof: For any -∞ < a,b,c < ∞, using the independency between X‾, Y‾, Z‾ and central limit theorem,

,

Therefore, by the definition of weak convergence (w.c.),

Function is not hard to see to be a continuous function. According to the property of weakly convergence,

,

For i∈{1,2,3} , by central limit theorem, in distribution. The chi square distribution has mean n_i-1 and variance 2_ni-2. Hence, in distribution. From Lemma 1, s_i converges in probability to σ_i because mean converges to 1 and variance converges to 0. From that, we have

Apply Slutsky Theorem to the above two convergences,

QED.

Proposition 2: X,Y,Z are independent distributions sharing the same mean and they have standard deviations 0 <σ ₁, σ₂, σ₃ < ∞. Draw n₁, n₂, n₃ samples from each of them independently, let n=n₁+n₂+n₃, if

,

Denote s₁, s₂, s₃ to be the sample variance of the samples and

,

Then when σ₁ = σ₂ = σ₃ and R₁ = R₂ = R₃

and

Proof:
To prove the first convergence, simply cancel out the σ’s and R’s from the results of Proposition 1 given the assumption of R_i and σ_i.
The second convergence is trivial given the first convergence.

QED.

Now let’s calculate the top right cube of “Control=Control2” case. Other probabilities in the tables can be calculated in a similar way via Proposition 2:

Define

From Proposition 2

Therefore

,

where φ is the density function of standard normal distribution

.