Initially, we defined expected value separately for discrete distributions, continuous distributions, and mixed distributions, in each case using density functions. Later we showed how these definitions can be unified, by first defining expected value for nonnegative random variables in terms of the right-tail distribution function. However, by far the best and most elegant definition of expected value is as an integral with respect to the underlying probability measure. This definition and a review of the properties of expected value are the goals of this section. No proofs are necessary (you will be happy to know), since all of the results follow from the general theory of integration. If you are a new student of probability, or are not interested in the measure-theoretic detail of the subject, you can safely skip this section.

Definitions

As usual, our starting point is a random experiment, as described in random experiment, modeled by a probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$. So $\mathrm{\Omega }$ is the set of outcomes, $\mathcal{F}$ is the $\sigma$-algebra of events, and $\mathbb{P}$ is the probability measure on the sample space $(\mathrm{\Omega },\mathcal{F})$.

Recall that a random variable $X$ for the experiment is simply a measurable function from $(\mathrm{\Omega },\mathcal{F})$ into another measurable space $(S,\mathcal{S})$. When $S\subseteq {\mathbb{R}}^n$, we assume that $S$ is Lebesgue measurable, and we take $\mathcal{S}$ to the $\sigma$-algebra of Lebesgue measurable subsets of $S$. As noted above, here is the measure-theoretic definition:

Let's review how the integral is defined in stages, but now using the notation of probability theory.

Thus, as with integrals generally, an expected value can exist as a number in $\mathbb{R}$ (in which case $X$ is integrable), can exist as $\mathrm{\infty }$ or $-\mathrm{\infty }$, or can fail to exist. In reference to part (a), a random variable with a finite set of values in $\mathbb{R}$ is a simple function in the terminology of general integration. In reference to part (b), note that the expected value of a nonnegative random variable always exists in $[0,\mathrm{\infty }]$. In reference to part (c), $\mathbb{E}(X)$ exists if and only if either $\mathbb{E}\left(X^{+}\right)<\mathrm{\infty }$ or $\mathbb{E}\left(X^{-}\right)<\mathrm{\infty }$.

Our next goal is to restate the basic theorems and properties of integrals, but in the notation of probability. Unless otherwise noted, all random variables are assumed to be real-valued.

Basic Properties

The Linear Properties

Perhaps the most important and basic properties are the linear properties. Part (a) is the additive property and part (b) is the scaling property.

Thus, part (a) holds if at least one of the expected values on the right is finite, or if both are $\mathrm{\infty }$, or if both are $-\mathrm{\infty }$. What is ruled out are the two cases where one expected value is $\mathrm{\infty }$ and the other is $-\mathrm{\infty }$, and this is what is meant by the indeterminate form $\mathrm{\infty }-\mathrm{\infty }$.

Equality and Order

Our next set of properties deal with equality and order. First, the expected value of a random variable over a null set is 0.

Random variables that are equivalent have the same expected value

Our next result is the positive property of expected value.

So, if $X$ is a nonnegative random variable then $\mathbb{E}(X)>0$ if and only if $\mathbb{P}(X>0)>0$. The next result is the increasing property of expected value, perhaps the most important property after linearity.

So if $X\le Y$ with probability 1 then, except in the two cases mentioned, $\mathbb{E}(X)<\mathbb{E}(Y)$ if and only if $\mathbb{P}(X<Y)>0$. The next result is the absolute value inequality.

Change of Variables and Density Functions

The Change of Variables Theorem

Suppose now that $X$ is a general random variable on the probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$, taking values in a measurable space $(S,\mathcal{S})$. Recall that the probability distribution of $X$ is the probability measure $P$ on $(S,\mathcal{S})$ given by $P(A)=\mathbb{P}(X\in A)$ for $A\in \mathcal{S}$. This is a special case of a new positive measure induced by a given positive measure and a measurable function. If $g:S\to \mathbb{R}$ is measurable, then $g(X)$ is a real-valued random variable. The following result shows how to computed the expected value of $g(X)$ as an integral with respect to the distribution of $X$, and is known as the change of variables theorem.

So, using the original definition and the change of variables theorem, and giving the variables explicitly for emphasis, we have $$\mathbb{E}[g(X)]={\int }_{\mathrm{\Omega }}g[X(\omega )]d\mathbb{P}(\omega )={\int }_{S}g(x)dP(x)$$

The Radon-Nikodym Theorem

Suppose now $\mu$ is a positive measure on $(S,\mathcal{S})$, and that the distribution of $X$ is absolutely continuous with respect to $\mu$. Recall that this means that $\mu (A)=0$ implies $P(A)=\mathbb{P}(X\in A)=0$ for $A\in \mathcal{S}$. By the Radon-Nikodym theorem, named for Johann Radon and Otto Nikodym, $X$ has a probability density function $f$ with respect to $\mu$. That is, $$P(A)=\mathbb{P}(X\in A)={\int }_{A}fd\mu ,A\in \mathcal{S}$$ In this case, we can write the expected value of $g(X)$ as an integral with respect to the probability density function.

Again, giving the variables explicitly for emphasis, we have the following chain of integrals: $$\mathbb{E}[g(X)]={\int }_{\mathrm{\Omega }}g[X(\omega )]d\mathbb{P}(\omega )={\int }_{S}g(x)dP(x)={\int }_{S}g(x)f(x)d\mu (x)$$

There are two critically important special cases.

Discrete Distributions

Suppose first that $(S,\mathcal{S},\mathrm{\#})$ is a discrete measure space, so that $S$ is countable, $\mathcal{S}=\mathcal{P}(S)$ is the collection of all subsets of $S$, and $\mathrm{\#}$ is counting measure on $(S,\mathcal{S})$. Thus, $X$ has a discrete distribution on $S$, and this distribution is always absolutely continuous with respect to $\mathrm{\#}$. Specifically, $\mathrm{\#}(A)=0$ if and only if $A=\mathrm{\varnothing }$ and of course $\mathbb{P}(X\in \mathrm{\varnothing })=0$. The probability density function $f$ of $X$ with respect to $\mathrm{\#}$, as we know, is simply $f(x)=\mathbb{P}(X=x)$ for $x\in S$. Moreover, integrals with respect to $\mathrm{\#}$ are sums, so $$\mathbb{E}[g(X)]=\sum _{x\in S}g(x)f(x)$$ assuming that the expected value exists. Existence in this case means that either the sum of the positive terms is finite or the sum of the negative terms is finite, so that the sum makes sense (and in particular does not depend on the order in which the terms are added). Specializing further, if $X$ itself is real-valued and $g=1$ we have $$\mathbb{E}(X)=\sum _{x\in S}xf(x)$$ which was our original definition of expected value in the discrete case.

Continuous Distributions

For the second special case, suppose that $(S,\mathcal{S},{\lambda }^n)$ is a Euclidean measure space, so that $S$ is a Lebesgue measurable subset of ${\mathbb{R}}^n$ for some $n\in {\mathbb{N}}_{+}$, $\mathcal{S}$ is the $\sigma$-algebra of Lebesgue measurable subsets of $S$, and ${\lambda }^n$ is Lebesgue measure on $(S,\mathcal{S})$. The distribution of $X$ is absolutely continuous with respect to ${\lambda }^n$ if ${\lambda }^n(A)=0$ implies $\mathbb{P}(X\in A)=0$ for $A\in \mathcal{S}$. If this is the case, then a probability density function $f$ of $X$ has its usual meaning. Thus, $$\mathbb{E}[g(X)]={\int }_{S}g(x)f(x)d{\lambda }^n(x)$$ assuming that the expected value exists. When $g$ is a typically nice function, this integral reduces to an ordinary $n$-dimensional Riemann integral of calculus. Specializing further, if $X$ is itself real-valued and $g=1$ then $$\mathbb{E}(X)={\int }_{S}xf(x)dx$$ which was our original definition of expected value in the continuous case.

Interchange Properties

In this subsection, we review properties that allow the interchange of expected value and other operations: limits of sequences, infinite sums, and integrals. We assume again that the random variables are real-valued unless otherwise specified.

Limits

Our first set of convergence results deals with the interchange of expected value and limits. We start with the expected value version of Fatou's lemma, named in honor of Pierre Fatou. Its usefulness stems from the fact that no assumptions are placed on the random variables, except that they be nonnegative.

Our next set of results gives conditions for the interchange of expected value and limits.

Statements about the random variables in [11] (nonnegative, increasing, existence of limit, etc.) need only hold with probability 1. Part (a) is the monotone convergence theorem, one of the most important convergence results and in a sense, essential to the definition of the integral in the first place. Parts (b) and (c) are slight generalizations of the monotone convergence theorem. In parts (a), (b), and (c), note that $\underset{n\to \mathrm{\infty }}{lim}{X}_n$ exists (with probability 1), although the limit may be $\mathrm{\infty }$ in parts (a) and (b) and $-\mathrm{\infty }$ in part (c) (with positive probability). Part (d) is the dominated convergence theorem, another of the most important convergence results. It's sometimes also known as Lebesgue's dominated convergence theorem in honor of Henri Lebesgue. Part (e) is a corollary of the dominated convergence theorem, and is known as the bounded convergence theorem.

Infinite Series

Our next results involve the interchange of expected value and an infinite sum, so these results generalize the basic additivity property of expected value.

Part (a) is a consequence of the monotone convergence theorem, and part (b) is a consequence of the dominated convergence theorem. In (b), note that $\sum _{n=1}^{\mathrm{\infty }}\left|X_n\right|<\mathrm{\infty }$ and hence $\sum _{n=1}^{\mathrm{\infty }}{X}_n$ is absolutely convergent with probability 1. Our next result is the additivity of the expected value over a countably infinite collection of disjoint events.

Of course, the previous theorem applies in particular if $X$ is nonnegative.

Integrals

Suppose that $(T,\mathcal{T},\mu )$ is a $\sigma$-finite measure space, and that $X_t$ is a real-valued random variable for each $t\in T$. Thus we can think of $\{X_t:t\in T\}$ is a stochastic process indexed by $T$. We assume that $(\omega ,t)\mapsto X_t(\omega )$ is measurable, as a function from the product space $(\mathrm{\Omega }\times T,\mathcal{F}\otimes \mathcal{T})$ into $\mathbb{R}$. Our next result involves the interchange of expected value and integral, and is a consequence of Fubini's theorem, named for Guido Fubini.

Fubini's theorem actually states that the two iterated integrals above equal the joint integral $${\int }_{\mathrm{\Omega }\times T}{X}_t(\omega )d(\mathbb{P}\otimes \mu )(\omega ,t)$$ where of course, $\mathbb{P}\otimes \mu$ is the product measure on $(\mathrm{\Omega }\times T,\mathcal{F}\otimes \mathcal{T})$. However, our interest is usually in evaluating the iterated integral above on the left in terms of the iterated integral on the right. Part (a) is the expected value version of Tonelli's theorem, named for Leonida Tonelli.

Examples and Exercises

You may have worked some of the computational exercises before, but try to see them in a new light, in terms of the general theory of integration.

The Cauchy Distribution

Recall that the Cauchy distribution, named for Augustin Cauchy, is a continuous distribution with probability density function $f$ given by $$f(x)=\frac{1}{\pi (1+x^2)},x\in \mathbb{R}$$

The Pareto Distribution

Recall that the Pareto distribution, named for Vilfredo Pareto, is a continuous distribution with probability density function $f$ given by $$f(x)=\frac{a}{x^{a+1}},x\in [1,\mathrm{\infty })$$ where $a>0$ is the shape parameter.

Special Results for Nonnegative Variables

For a nonnegative variable, the moments can be obtained from integrals of the right-tail distribution function.

When $n=1$ we have $\mathbb{E}(X)={\int }_0^{\mathrm{\infty }}\mathbb{P}(X>x)dx$. We saw this result before, but now we can understand the proof in terms of Fubini's theorem.

For a random variable taking nonnegative integer values, the moments can be computed from sums involving the right-tail distribution function.

When $n=1$ we have $\mathbb{E}(X)=\sum _{k=0}^{\mathrm{\infty }}\mathbb{P}(X\ge k)$. We also saw this result before, but now we can understand the proof in terms of the interchange of sum and expected value.