The Law of Large Numbers for Independent Repeated Bernoulli Trials

Consider an experiment with two possible outcomes, denoted by success and failure. Suppose, however, that the probability of success at each trial is unknown. According to the frequency interpretation of probability, represents the relative frequency of successes in an indefinitely prolonged series of trials. Consequently, one might think that in order to determine one must only perform a long series of trials and take as the value of the observed relative frequency of success. The question arises: can one justify this procedure, not by appealing to the frequency interpretation of probability theory, but by appealing to the mathematical theory of probability?

The mathematical theory of probability is a logical construct, consisting of conclusions logically deduced from the axioms of probability theory. These conclusions are applicable to the world of real experience in the sense that they are conclusions about real phenomena, which are assumed to satisfy the axioms. We now show that one can reach a conclusion within the mathematical theory of probability that may be interpreted to justify the frequency interpretation of probability (and consequently may be used to justify the procedure described for estimating ). This result is known as the law of large numbers, since it applies to the outcome of a large number of trials. The law of large numbers we are about to investigate may be considerably generalized. Consequently, the version to be discussed is called the Bernoulli law of large numbers , as it was first discovered by Jacob Bernoulli and published in his posthumous book Ars conjectandi (1713).

The Bernoulli Law of Large Numbers . Let be the observed number of successes in independent repeated Bernoulli trials, with probability of success at each trial. Let

denote the relative frequency of successes in the trials. Then, for any positive number , no matter how small, it follows that

In words, (5.2) and (5.3) state that as the number of trials tends to infinity the relative frequency of successes in trials tends to the true probability of success at each trial, in the probabilistic sense that any nonzero difference between and becomes less and less probable of observation as the number of trials is increased indefinitely.

Bernoulli proved (5.3) by a tedious evaluation of the probability in (5.3). Using Chebyshev’s inequality, one can give a very simple proof of (5.3). By using the fact that the probability law of has mean and variance , one may prove that the probability law of has mean and variance . Consequently, for any

Now, for any value of in the interval

using the fact that . Consequently, for any

no matter what the true value of . To prove (5.2) , one uses (5.3) and the fact that

It is shown in section 5 of Chapter 8 that the foregoing method of proof, using Chebyshev’s inequality, permits one to prove that if , is a sequence of independent observations of a numerical valued random phenomenon whose probability law has mean then for any

The result given by (5.8) is known as the law of large numbers.

The Bernoulli law of large numbers states that to estimate the unknown value of , as an estimate of , the observed relative frequency of successes in trials can be employed; this estimate becomes perfectly correct as the number of trials becomes infinitely large. In practice, a finite number of trials is performed. Consequently, the number of trials must be determined, in order that, with high probability, the observed relative frequency be within a preassigned distance from . In symbols, to any number one desires to find so that

where we write to indicate that the probability is being calculated under the assumption that is the true probability of success at each trial.

One may obtain an expression for the value of that satisfies (5.9) by means of Chebyshev’s inequality. Since

it follows that (5.9) is satisfied if is chosen so that

The estimate of given by (5.11) can be improved upon. In section 2 of Chapter 6 we prove the normal approximation to the binomial law. In particular, it is shown that if is the probability of success at each trial then the number of successes in independent repeated Bernoulli trials approximately satisfies, for any , Consequently, the relative frequency of successes satisfies, for any , To obtain (5.13) from (5.12), let .

Define as the solution of the equation

A table of selected values of is given in Table 5A.

From (5.13) we may obtain the conclusion that

To justify (5.15), note that implies that the right-hand side of (5.13) is greater than the left-hand side of (5.14).

Since for all , we finally obtain from (5.15) that (5.9) will hold if

Exercises