Conjugate prior

In Bayesian probability theory, if, given a likelihood function

p

(

x

∣

θ

)

{\displaystyle p(x\mid \theta )}

, the posterior distribution

p

(

θ

∣

x

)

{\displaystyle p(\theta \mid x)}

is in the same probability distribution family as the prior probability distribution

p

(

θ

)

{\displaystyle p(\theta )}

, the prior and posterior are then called conjugate distributions with respect to that likelihood function and the prior is called a conjugate prior for the likelihood function

p

(

x

∣

θ

)

{\displaystyle p(x\mid \theta )}

.

A conjugate prior is an algebraic convenience, giving a closed-form expression for the posterior; otherwise, numerical integration may be necessary. Further, conjugate priors may clarify how a likelihood function updates a prior distribution.

The concept, as well as the term "conjugate prior", were introduced by Howard Raiffa and Robert Schlaifer in their work on Bayesian decision theory. A similar concept had been discovered independently by George Alfred Barnard.

Example

The form of the conjugate prior can generally be determined by inspection of the probability density or probability mass function of a distribution. For example, consider a random variable which consists of the number of successes

s

{\displaystyle s}

in

n

{\displaystyle n}

Bernoulli trials with unknown probability of success

q

{\displaystyle q}

in [0,1]. This random variable will follow the binomial distribution, with a probability mass function of the form

p

(

s

)

=

(

n

s

)

q

s

(

1

−

q

)

n

−

s

{\displaystyle p(s)={n \choose s}q^{s}(1-q)^{n-s}}

The usual conjugate prior is the beta distribution with parameters (

α

{\displaystyle \alpha }

,

β

{\displaystyle \beta }

):

p

(

q

)

=

q

α

−

1

(

1

−

q

)

β

−

1

B

(

α

,

β

)

{\displaystyle p(q)={q^{\alpha -1}(1-q)^{\beta -1} \over \mathrm {B} (\alpha ,\beta )}}

where

α

{\displaystyle \alpha }

and

β

{\displaystyle \beta }

are chosen to reflect any existing belief or information (

α

=

1

{\displaystyle \alpha =1}

and

β

=

1

{\displaystyle \beta =1}

would give a uniform distribution) and

B

(

α

,

β

)

{\displaystyle \mathrm {B} (\alpha ,\beta )}

is the Beta function acting as a normalising constant.

In this context,

α

{\displaystyle \alpha }

and

β

{\displaystyle \beta }

are called hyperparameters (parameters of the prior), to distinguish them from parameters of the underlying model (here

q

{\displaystyle q}

). A typical characteristic of conjugate priors is that the dimensionality of the hyperparameters is one greater than that of the parameters of the original distribution.