Conditional Adversarial Domain Adaptation的理论支持

Reproducing Kernel Hilbert Space

Given a nonempty set $\mathcal{X}$ , and $\mathcal{H}$ is a Hilbert space of function $f:\mathcal{X}\to R$ Then $\mathcal{H}$ called a Reproducing Kernel Hilbert Space endowed with the dot product $<\cdot ,\cdot >$, if there exists a function $k:\mathcal{X}\times \mathcal{X}\to R$ with the following properties:

1. $<f(\cdot ),k(x,\cdot )>=f(x)$ , in particular $<k(x,\cdot ),k(x^{\prime },\cdot )>=k(x,x^{\prime })$
2. k spans $\mathcal{H}$ : H = s p a n { k ( x , ⋅ ) ∣ x ∈ X } = { f ( ⋅ ) = ∑ i = 1 m α i k ( x , ⋅ ) : m ∈ N , x i ∈ X , α i ∈ R } \mathcal{H}= span\{ k(x,\cdot) | x \in \mathcal{X} \}=\{ f(\cdot)= \sum_{i=1}^m \alpha_ik(x,\cdot) : m \in N , x_i \in \mathcal{X} , \alpha_i \in R\} H=span{k(x,⋅)∣x∈X}={f(⋅)=∑i=1m​αi​k(x,⋅):m∈N,xi​∈X,αi​∈R}

Hilbert Space Embedding

$\mathcal{X}$ : the domain of observations
${\mathbf{P}}_x$ : a probability measure on $\mathcal{X}$

$\mathcal{Y}$ : the domain of observations
${\mathbf{P}}_y$ : a probability measure on $\mathcal{Y}$.
${\mathbf{P}}_{x\times y}$ : A joint probability measure on $\mathcal{X}\times \mathcal{Y}$

reproducing kernel Hilbert space (RKHS) $\mathcal{H}$ of functions on $\mathcal{X}$ with kernel $k(x,x^{\prime }):=<\phi (x),\phi (x^{\prime })>$.

mean map $\mu$ :
$\mu [{\mathbf{P}}_x]:={\mathbf{E}}_x[k(x,\cdot )]$
$\mu [X]:=\frac{1}{m}{\sum }_{i=1}^{m}k(x_i,\cdot )$

then $\mu [{\mathbf{P}}_x]$ is an element of th Hilbert space ,so
$<\mu [{\mathbf{P}}_x],f>={\mathbf{E}}_x[f(x)]$
$<\mu [X],f>=\frac{1}{m}{\sum }_{i=1}^{m}f(x_i)$
where X = { x 1 , x 2 , ⋯   , x m } X=\{ x_1 ,x_2 , \cdots , x_m \} X={x1​,x2​,⋯,xm​} is assumed to be drawn independently and identically distributed from ${\mathbf{P}}_x$. and $\mu [X]$ is an estimate of the mean map.
$\mu [X]=\frac{1}{m}{\sum }_{i=1}^{m}k(x_i,\cdot )=\frac{1}{m}{\gamma 1}_m$ , where γ : = { k ( x 1 , ⋅ ) , k ( x 2 , ⋅ ) , ⋯   , k ( x m , ⋅ ) } \mathbf{\gamma} :=\{ k(x_1,\cdot) , k(x_2,\cdot) , \cdots , k(x_m,\cdot)\} γ:={k(x1​,⋅),k(x2​,⋅),⋯,k(xm​,⋅)}

Covariance operators

Given A joint probability measure ${\mathbf{P}}_{x\times y}$ on $\mathcal{X}\times \mathcal{Y}$ , the uncentered covariance operator ${\mathcal{C}}_{XY}$ (Baker, 1973) $${\mathcal{C}}_{XY}:={\mathbb{E}}_{XY}[\phi (x)\otimes \varphi (y)]$$, where $\otimes$ denotes tensor product.

Given $m$ pairs of $i.i.d.$ observations { ( x l , y l ) } l = 1 m \{ (x^l , y^l) \}_{l=1}^m {(xl,yl)}l=1m​ , we denote by $\gamma =(\phi (x^1),\phi (x^2),\cdots ,\phi (x^m))$ and $\Phi =(\varphi (y^1),\varphi (y^2),\cdots ,\varphi (y^m))$. Conceptually, the covariance operator ${\mathcal{C}}_{XY}$ can then be estimated as ${\hat{\mathcal{C}}}_{XY}=\frac{1}{m}\gamma {\Phi }^T$
Notes on mean embeddings and covariance
operators

tensor product :Notes on Tensor Products and the Exterior Algebra

Tensor Product Kernels: Characteristic Propertyand Universality
张量积计算
由泛函分析得
let $(X,<\cdot ,\cdot >)$ be an inner product space over a field $\mathbb{F}$. For each $x\in X$ , define$$\Vert x\Vert :=\sqrt{<x,x>}$$ Then $\Vert \cdot \Vert$ defines a norm on $X$. That is , $(X,\Vert \cdot \Vert )$ is a normed linear space over $\mathbb{F}$.

问题：
原论文提到

Conditional embedding operators

By analogy with the embedding of marginal distributions, the conditional density $\mathbb{P}(Y|x)$ an also be rep-resented as an RKHS element: $$\mu [Y|x]:={\mathbb{E}}_{Y|x}[\varphi (Y)]$$with each element corresponding toa particular value of $x$.
These conditional embeddings can be defined via a conditional embedding operator ${\mathcal{C}}_{Y|X}:\mathcal{F}\to \mathcal{G}$
$\mu [Y|x]={\mathcal{C}}_{Y|X}\phi (x):={\mathcal{C}}_{YX}{\mathcal{C}}_{XX}^{-1}\phi (x)$

Given $m$ pairs of $i.i.d.$ observations { ( x l , y l ) } l = 1 m \{ (x^l , y^l) \}_{l=1}^m {(xl,yl)}l=1m​ from ${\mathbb{P}}_{x\times y}$, the conditional embedding operator can be estimated as $${\hat{\mathcal{C}}}_{Y|X}=\frac{\Phi {\gamma }^T}{m}(\frac{\gamma {\gamma }^T}{m}+\lambda I{)}^{-1}=\Phi (K+\lambda mI{)}^{-1}{\gamma }^T$$where $K:={\gamma }^T\gamma$ with $(i,j)$th entry $k(x_i,x_j)$

参考文献

A Hilbert Space Embedding for Distributions PDF
Hilbert Space Embeddings of Hidden Markov Models
Hilbert Space Embeddings of Hidden Markov Models ppt

Generalization and Equilibrium in Generative Adversarial Nets (GANs)

• class of generators : { G u , u ∈ U } \{ G_u , u \in \mathcal{U} \} {Gu​,u∈U} , where $G$ is a function : ${\mathbb{R}}^l\to {\mathbb{R}}^d$,and $u$ is parameters of the generators.
• $x=G_u(h),h\sim l$-dimensional spherical Gaussian distribution
• class of discriminators : { D v , v ∈ V } \{ D_v , v\in \mathcal{V} \} {Dv​,v∈V} , where $D$ is a function : ${\mathbb{R}}^d\to [0,1]$
• $D_v(x)$ is usually interpreted as the probability that the sample $x$ comes from the real distribution $D_{real}$

Objective functions $$\underset{u\in \mathcal{U}}{\min }\underset{v\in \mathcal{V}}{\max }{\mathbb{E}}_{x\in {\mathcal{D}}_{real}}[\log D_v(x)]+{\mathbb{E}}_{x\in {\mathcal{D}}_{G_u}}[\log (1-D_v(x))]$$

原作者的解读 知乎
最新|哥伦比亚机器学习研讨会：GAN理论/应用最新进展总结（附视频）
混合高斯分布