Basic Theory

As usual, our starting point is a random experiment with an underlying sample space and a probability measure $\mathbb{P}$. In the basic statistical model, we have an observable random variable $\mathit{X}$ taking values in a set $S$. In general, $\mathit{X}$ can have quite a complicated structure. For example, if the experiment is to sample $n$ objects from a population and record various measurements of interest, then $$\mathit{X}=(X_1,X_2,\dots ,X_n)$$ where $X_i$ is the vector of measurements for the $i$th object.

Suppose also that the distribution of $\mathit{X}$ depends on a parameter $\theta$ with values in a set $T$. The parameter may also be vector valued, in which case $T\subseteq {\mathbb{R}}^k$ for some $k\in {\mathbb{N}}_{+}$ and the parameter has the form $\mathit{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_k)$.

The Bayesian Formulation

Recall that in Bayesian analysis, named for the infamous Thomas Bayes, the unknown parameter $\theta$ is treated as the observed value of a random variable $\mathrm{\Theta }$ with values in $T$. Here is a brief review:

The prior distribution is often subjective, and is chosen to reflect our knowledge, if any, of the parameter. In some cases, we can recognize the posterior distribution from the functional form of $\theta \mapsto h(\theta )f(\mathit{x}\mid \theta )$ without having to actually compute the normalizing constant $f(\mathit{x})$, and thus reducing the computational burden significantly. In particular, this is often the case when we have a conjugate parametric family of distributions of $\mathrm{\Theta }$. Recall that this means that when the prior distribution of $\mathrm{\Theta }$ belongs to the family, so does the posterior distribution of $\mathrm{\Theta }$ given $\mathit{X}=\mathit{x}\in S$.

The most important special case arises when we have a basic variable $X$ with values in a set $R$, and given $\mathrm{\Theta }=\theta \in T$, the data vector $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from $X$. That is, given $\mathrm{\Theta }=\theta \in T$, $\mathit{X}$ is a sequence of independent, identically distributed variables, each with the same distribution as $X$ given $\mathrm{\Theta }=\theta$. Thus $S=R^n$ and if $X$ has conditional probability density function $g(x\mid \theta )$, then $$f(\mathit{x}\mid \theta )=g(x_1\mid \theta )g(x_2\mid \theta )\cdots g(x_n\mid \theta ),\mathit{x}=(x_1,x_2,\dots ,x_n)\in S$$

Confidence Sets

Now let $C(\mathit{X})$ be a confidence set (that is, a subset of the parameter set $T$ that depends on the data variable $\mathit{X}$ but no unknown parameters).

In defintion [2], only $\mathrm{\Theta }$ is random and thus the probability above is computed using the posterior probability density function $\theta \mapsto h(\theta \mid \mathit{x})$.

In definition [3], $\mathit{X}$ and $\mathrm{\Theta }$ are both random, and so the probability above would be computed using the joint probability density function $(\mathit{x},\theta )\mapsto h(\theta )f(\mathit{x}\mid \theta )$. Whatever the philosophical arguments may be, definition [2] is certainly the easier one from a computational viewpoint, and hence is the one most commonly used.

Let us compare the classical and Bayesian approaches. In the classical approach, the parameter $\theta$ is deterministic, but unknown. Before the data are collected, the confidence set $C(\mathit{X})$ (which is random by virtue of $\mathit{X}$) will contain the parameter with probability $1-\alpha$. After the data are collected, the computed confidence set $C(\mathit{x})$ either contains $\theta$ or does not, and we will usually never know which. By contrast in a Bayesian confidence set, the random parameter $\mathrm{\Theta }$ falls in the computed, deterministic confidence set $C(\mathit{x})$ with probability $1-\alpha$.

As in past sections, $r$ is the fraction of $\alpha$ in the right tail of the posterior distribution and $1-r$ is the fraction of $\alpha$ in the left tail of the posterior distribution. As usual, $r=\frac{1}{2}$ gives the symmetric, two-sided confidence interval; letting $r\to 0$ gives the confidence lower bound; and letting $r\to 1$ gives the confidence upper bound.

Applications

The Bernoulli Distribution

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the Bernoulli distribution with unknown success parameter $p\in (0,1)$. In the usual language of reliability, $X_i=1$ means success on trial $i$ and $X_i=0$ means failure on trial $i$. The distribution is named for Jacob Bernoulli. Recall that the Bernoulli distribution has probability density function (given $p$) $$g(x\mid p)=p^x(1-p{)}^{1-x},x\in \{0,1\}$$ Note that the number of successes in the $n$ trials is $Y=\sum _{i=1}^n{X}_i$. Given $p$, random variable $Y$ has the binomial distribution with parameters $n$ and $p$.

In our previous discussion of Bayesian estimation, we modeled the parameter $p$ with a random variable $P$ that has a beta distribution. This family of disstributions is conjugate for $P$. Specifically, if the prior distribution of $P$ is beta with left parameter $a>0$ and right parameter $b>0$, then the posterior distribution of $P$ given $\mathit{X}$ is beta with left parameter $a+Y$ and right parameter $b+(n-Y)$; the left parameter is increased by the number of successes and the right parameter by the number of failure. It follows that a $1-\alpha$ level Bayesian confidence interval for $p$ is $[U_{\alpha /2}(y),U_{1-\alpha /2}(y)]$ where $U_r(y)$ is the quantile of order $r$ for the posterior beta distribution. In the special case $a=b=1$ the prior distribution is uniform on $(0,1)$ and reflects a lack of previous knowledge about $p$.

The Poisson Distribution

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the Poisson distribution with parameter $\lambda \in (0,\mathrm{\infty })$. Recall that the Poisson distribution is often used to model the number of random points in a region of time or space, particularly in the contest of the Poisson process. The distribution is named for the inimitable Simeon Poisson and given $\lambda$, has probability density function $$g(x\mid \theta )=e^{-\lambda }\frac{{\lambda }^x}{x!},x\in \mathbb{N}$$ As usual, we will denote the sum of the sample values by $Y=\sum _{i=1}^n{X}_i$. Given $\lambda$, random variable $Y$ also has a Poisson distribution, but with parameter $n\lambda$.

In our previous discussion of Bayesian estimation, we showed modeled $\lambda$ with a random variable $\mathrm{\Lambda }$ that has a gamma distribution. This family of distributions is conjugate for $\mathrm{\Lambda }$. Specifically, if the prior distribution of $\mathrm{\Lambda }$ is gamma with shape parameter $k>0$ and rate parameter $r>0$ (so that the scale parameter is $1/r$), then the posterior distribution of $\mathrm{\Lambda }$ given $\mathit{X}$ is gamma with shape parameter $k+Y$ and rate parameter $r+n$. It follows that a $1-\alpha$ level Bayesian confidence interval for $\lambda$ is $[U_{\alpha /2}(y),U_{1-\alpha /2}(y)]$ where $U_p(y)$ is the quantile of order $p$ for the posterior gamma distribution.

The Normal Distribution

Suppose that $\mathit{x}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the normal distribution with unknown mean $\mu \in \mathbb{R}$ and known variance ${\sigma }^2\in (0,\mathrm{\infty })$. Of course, the normal distribution plays an especially important role in statistics, in part because of the central limit theorem. The normal distribution is widely used to model physical quantities subject to numerous small, random errors. Recall that the normal probability density function (given $\mu$) is $$g(x\mid \mu )=\frac{1}{\sqrt{2\pi }\sigma }\exp [-{\left(\frac{x-\mu }{\sigma }\right)}^2],x\in \mathbb{R}$$ We denote the sum of the sample values by $Y=\sum _{i=1}^n{X}_i$. Recall that $Y$ also has a normal distribution (given $\mu$), but with mean $n\mu$ and variance $n{\sigma }^2$.

In our previous discussion of Bayesian estimation, we modeled $\mu$ with a random variable $\mathrm{\Psi }$ that also has a normal distribution. This family is conjugate for $\mathrm{\Psi }$ (with $\sigma$ known). Specifically, if the prior distribution of $\mathrm{\Psi }$ is normal with mean $a\in \mathbb{R}$ and standard deviation $b\in (0,\mathrm{\infty })$, then the posterior distribution of $\mathrm{\Psi }$ given $\mathit{X}$ is also normal, with $$\mathbb{E}(\mu \mid \mathit{X})=\frac{Y{b}^2+a{\sigma }^2}{{\sigma }^2+n{b}^2},\text{var}(\mu \mid \mathit{X})=\frac{{\sigma }^2{b}^2}{{\sigma }^2+n{b}^2}$$ It follows that a $1-\alpha$ level Bayesian confidence interval for $\mu$ is $[U_{\alpha /2}(y),U_{1-\alpha /2}(y)]$ where $U_p(y)$ is the quantile of order $p$ for the posterior normal distribution. An interesting special case is when $b=\sigma$, so that the standard deviation of the prior distribution of $\mu$ is the same as the standard deviation of the sampling distribution. In this case, the posterior mean is $(Y+a)/(n+1)$ and the posterior variance is ${\sigma }^2/(n+1)$

The Beta Distribution

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the beta distribution with unknown left shape parameter $a\in (0,\mathrm{\infty })$ and right shape parameter $b=1$. The beta distribution is widely used to model random proportions and probabilities and other variables that take values in bounded intervals. Recall that the probability density function (given $a$) is $$g(x\mid a)=a{x}^{a-1},x\in (0,1)$$ We denote the product of the sample values by $W=X_1{X}_2\cdots X_n$.

In our previous discussion of Bayesian estimation, we modeled $a$ with a random variable $A$ that has a gamma distribution. This family of distributions is conjugate for $a$. Specifically, if the prior distribution of $A$ is gamma with shape parameter $k>0$ and rate parameter $r>0$, then the posterior distribution of $A$ given $\mathit{X}$ is also gamma, with shape parameter $k+n$ and rate parameter $r-\ln (W)$. It follows that a $1-\alpha$ level Bayesian confidence interval for $A$ is $[U_{\alpha /2}(w),U_{1-\alpha /2}(w)]$ where $U_p(w)$ is the quantile of order $p$ for the posterior gamma distribution. In the special case that $k=1$, the prior distribution of $a$ is exponential with rate parameter $r$.

The Pareto Distribution

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the Pareto distribution with shape parameter $a\in (0,\mathrm{\infty })$ and scale parameter $b=1$. The Pareto distribution is used to model certain financial variables and other variables with heavy-tailed distributions, and is named for Vilfredo Pareto. Recall that the probability density function (given $a$) is $$g(x\mid a)=\frac{a}{x^{a+1}},x\in [1,\mathrm{\infty })$$ We denote the product of the sample values by $W=X_1{X}_2\cdots X_n$.

In our previous discussion of Bayesian estimation, we showed modeled $a$ with a random variable $A$ that has a gamma distribution. This family of distributions is conjugate for $A$. Specifically, if the prior distribution of $A$ is gamma with shape parameter $k>0$ and rate parameter $r>0$, then the posterior distribution of $A$ given $\mathit{X}$ is also gamma, with shape parameter $k+n$ and rate parameter $r+\ln (W)$. It follows that a $1-\alpha$ level Bayesian confidence interval for $a$ is $[U_{\alpha /2}(w),U_{1-\alpha /2}(w)]$ where $U_p(w)$ is the quantile of order $p$ for the posterior gamma distribution. In the special case that $k=1$, the prior distribution of $a$ is exponential with rate parameter $r$.