The Problem of Addition of Independent Random Variables

A large number of the problems which arise in applications of probability theory may be regarded as special cases of the following general problem, which we call the problem of addition of independent random variables; find, either exactly or approximately, the probability law of a random variable that arises as the sum of independent random variables , , whose joint probability law is known . The fundamental role played by this problem in probability theory is best described by a quotation from an article by Harald Cramér, “Problems in Probability Theory”, Annals of Mathematical Statistics , Volume 18 (1947), p. 169.

During the early development of the theory of probability, the majority of problems considered were connected with gambling. The gain of a player in a certain game may be regarded as a random variable, and his total gain in a sequence of repetitions of the game is the sum of a number of independent variables, each of which represents the gain in a single performance of the game. Accordingly, a great amount of work was devoted to the study of the probability distributions of such sums. A little later, problems of a similar type appeared in connection with the theory of errors of observation, when the total error was considered as the sum of a certain number of partial errors due to mutually independent causes. At first, only particular cases were considered; but gradually general types of problems began to arise, and in the classical work of Laplace several results are given concerning the general problem to study the distribution of a sum

of independent variables, when the distributions of the are given. This problem may be regarded as the very starting point of a large number of those investigations by which the modern Theory of Probability was created. The efforts to prove certain statements of Laplace, and to extend his results further in various directions, have largely contributed to the introduction of rigorous foundations of the subject, and to the development of the analytical methods.

In this chapter we discuss the methods and notions by which a precise formulation and solution is given to the problem of addition of independent random variables. To begin with, in this section we discuss the two most important ways in which this problem can arise, namely in the analysis of sample averages and in the analysis of random walks .

Sample Averages . We have defined a sample of size of a random variable as a set of jointly distributed random variables , whose individual probability laws coincide, for , with the probability law of ; in particular, the distribution function of coincides with the distribution function of . We have defined the sample as a random sample if the random variables are independent.

Given a sample of size of the random variable and any Borel function of a real variable, we define the sample average of , denoted by , as the arithmetic mean of the values , of the function at the members of the sample; in symbols,

Of special importance are the sample mean , defined by and the sample variance , defined by

For a given function the sample average is a random variable , for it is a function of the random variables . The value of will, in general, be different when it is computed on the basis of two different samples of size . The sample average , like any other random variable, has a mean , a variance , a distribution function , a moment-generating function , and, depending on whether it is a continuous or a discrete random variable, a probability density function or a probability mass function . Our aim in this chapter and the next is to develop techniques for computing these quantities, both exactly and approximately, and especially to study their behavior for large sample sizes. The reader who goes on to the study of mathematical statistics will find that these techniques provide the framework for many of the concepts of statistics.

To study sample averages with respect to a random sample, it suffices to consider the sum of independent random variables , since the random variables are independent if the random variables are. Thus it is seen that the study of sample averages has been reduced to the study of sums of independent random variables .

Random Walk. Consider a particle that at a certain time is located at the point 0 on a certain straight line. Suppose that it then suffers displacements along the straight line in the form of a series of steps, denoted by , in which, for any integer represents the displacement suffered by the particle at the th step. The size of the th step is assumed to be a random variable with a known probability law. The particle can thus be imagined as executing a random walk along the line, its position (denoted by ) after steps being the sum of the steps ; in symbols, . Clearly, is a random variable, and the problem of finding the probability law of naturally arises; in other words, one wishes to know, for any integer and any interval to , the probability that after steps the particle will lie between and , inclusive.

The problem of random walks can be generalized to two or more dimensions. Suppose that the particle at each stage suffers a displacement in an plane, and let and denote, respectively, the change in the - and -coordinates of the particle at the th step. The position of the particle after steps is given by the random 2-tuple , in which and . We now have the problem of determining the joint probability law of the random variables and .

The problem of random walks occurs in many branches of physics, especially in its 2-dimensional form. The eminent mathematical statistician, Karl Pearson, was the first to formulate explicitly the problem of the 2-dimensional random walk. After Pearson had formulated this problem in 1905, the renowned physicist, Lord Rayleigh, pointed out that the problem of random walks was formally “the same as that of the composition of isoperiodic vibrations of unit amplitude and of phases distributed at random”, which he had considered as early as 1880 (for this quotation and a history of the problem of random walks, see p. 87 of S. Chandrasekhar, “Stochastic Problems in Physics and Astronomy”, Reviews of Modern Physics , Volume 15 (1943), pp. 1–89). Almost all scattering problems in physics are instances of the problem of random walks.

In the study of sums of independent random variables a basic role is played by the notion of the characteristic function of a random variable. This notion is introduced in section 2.