Kernel-independent component analysis

In statistics, kernel-independent component analysis (kernel ICA) is an efficient algorithm for independent component analysis which estimates source components by optimizing a generalized variance contrast function, which is based on representations in a reproducing kernel Hilbert space. Those contrast functions use the notion of mutual information as a measure of statistical independence.

Main idea

Kernel ICA is based on the idea that correlations between two random variables can be represented in a reproducing kernel Hilbert space (RKHS), denoted by

F

{\displaystyle {\mathcal {F}}}

, associated with a feature map

L

x

:

F

↦

R

{\displaystyle L_{x}:{\mathcal {F}}\mapsto \mathbb {R} }

defined for a fixed

x

∈

R

{\displaystyle x\in \mathbb {R} }

. The

F

{\displaystyle {\mathcal {F}}}

-correlation between two random variables

X

{\displaystyle X}

and

Y

{\displaystyle Y}

is defined as

ρ

F

(

X

,

Y

)

=

max

f

,

g

∈

F

corr

⁡

(

⟨

L

X

,

f

⟩

,

⟨

L

Y

,

g

⟩

)

{\displaystyle \rho _{\mathcal {F}}(X,Y)=\max _{f,g\in {\mathcal {F}}}\operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle )}

where the functions

f

,

g

:

R

→

R

{\displaystyle f,g:\mathbb {R} \to \mathbb {R} }

range over

F

{\displaystyle {\mathcal {F}}}

and

corr

⁡

(

⟨

L

X

,

f

⟩

,

⟨

L

Y

,

g

⟩

)

:=

cov

⁡

(

f

(

X

)

,

g

(

Y

)

)

var

⁡

(

f

(

X

)

)

1

/

2

var

⁡

(

g

(

Y

)

)

1

/

2

{\displaystyle \operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle ):={\frac {\operatorname {cov} (f(X),g(Y))}{\operatorname {var} (f(X))^{1/2}\operatorname {var} (g(Y))^{1/2}}}}

for fixed

f

,

g

∈

F

{\displaystyle f,g\in {\mathcal {F}}}

. Note that the reproducing property implies that

f

(

x

)

=

⟨

L

x

,

f

⟩

{\displaystyle f(x)=\langle L_{x},f\rangle }

for fixed

x

∈

R

{\displaystyle x\in \mathbb {R} }

and

f

∈

F

{\displaystyle f\in {\mathcal {F}}}

. It follows then that the

F

{\displaystyle {\mathcal {F}}}

-correlation between two independent random variables is zero.

This notion of

F

{\displaystyle {\mathcal {F}}}

-correlations is used for defining contrast functions that are optimized in the Kernel ICA algorithm. Specifically, if

X

:=

(

x

i

j

)

∈

R

n

×

m

{\displaystyle \mathbf {X} :=(x_{ij})\in \mathbb {R} ^{n\times m}}

is a prewhitened data matrix, that is, the sample mean of each column is zero and the sample covariance of the rows is the

m

×

m

{\displaystyle m\times m}

dimensional identity matrix, Kernel ICA estimates a

m

×

m

{\displaystyle m\times m}

dimensional orthogonal matrix

A

{\displaystyle \mathbf {A} }

so as to minimize finite-sample

F

{\displaystyle {\mathcal {F}}}

-correlations between the columns of

S

:=

X

A

′

{\displaystyle \mathbf {S} :=\mathbf {X} \mathbf {A} ^{\prime }}

.