Dual total correlation

In information theory, dual total correlation, information rate, excess entropy, or binding information is one of several known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation.

Definition

For a set of n random variables

{

X

1

,

…

,

X

n

}

{\displaystyle \{X_{1},\ldots ,X_{n}\}}

, the dual total correlation

D

(

X

1

,

…

,

X

n

)

{\displaystyle D(X_{1},\ldots ,X_{n})}

is given by

D

(

X

1

,

…

,

X

n

)

=

H

(

X

1

,

…

,

X

n

)

−

∑

i

=

1

n

H

(

X

i

∣

X

1

,

…

,

X

i

−

1

,

X

i

+

1

,

…

,

X

n

)

,

{\displaystyle D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),}

where

H

(

X

1

,

…

,

X

n

)

{\displaystyle H(X_{1},\ldots ,X_{n})}

is the joint entropy of the variable set

{

X

1

,

…

,

X

n

}

{\displaystyle \{X_{1},\ldots ,X_{n}\}}

and

H

(

X

i

∣

⋯

)

{\displaystyle H(X_{i}\mid \cdots )}

is the conditional entropy of variable

X

i

{\displaystyle X_{i}}

, given the rest.

Normalized

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value

H

(

X

1

,

…

,

X

n

)

{\displaystyle H(X_{1},\ldots ,X_{n})}

,

N

D

(

X

1

,

…

,

X

n

)

=

D

(

X

1

,

…

,

X

n

)

H

(

X

1

,

…

,

X

n

)

.

{\displaystyle ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.}

Relationship with Total Correlation

Dual total correlation is non-negative and bounded above by the joint entropy

H

(

X

1

,

…

,

X

n

)

{\displaystyle H(X_{1},\ldots ,X_{n})}

.

0

≤

D

(

X

1

,

…

,

X

n

)

≤

H

(

X

1

,

…

,

X

n

)

.

{\displaystyle 0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).}

Secondly, Dual total correlation has a close relationship with total correlation,

C

(

X

1

,

…

,

X

n

)

{\displaystyle C(X_{1},\ldots ,X_{n})}

, and can be written in terms of differences between the total correlation of the whole, and all subsets of size

N

−

1

{\displaystyle N-1}

:

D

(

X

)

=

(

N

−

1

)

C

(

X

)

−

∑

i

=

1

N

C

(

X

−

i

)

{\displaystyle D({\textbf {X}})=(N-1)C({\textbf {X}})-\sum _{i=1}^{N}C({\textbf {X}}^{-i})}

where

X

=

{

X

1

,

…

,

X

n

}

{\displaystyle {\textbf {X}}=\{X_{1},\ldots ,X_{n}\}}

and

X

−

i

=

{

X

1

,

…

,

X

i

−

1

,

X

i

+

1

,

…

,

X

n

}

{\displaystyle {\textbf {X}}^{-i}=\{X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\}}

Furthermore, the total correlation and dual total correlation are related by the following bounds:

C

(

X

1

,

…

,

X

n

)

n

−

1

≤

D

(

X

1

,

…

,

X

n

)

≤

(

n

−

1

)

C

(

X

1

,

…

,

X

n

)

.

{\displaystyle {\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).}

Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:

Ω

(

X

)

=

C

(

X

)

−

D

(

X

)

{\displaystyle \Omega ({\textbf {X}})=C({\textbf {X}})-D({\textbf {X}})}

.