《Distributionally Robust Learning》第二章读书笔记

2 The Wasserstein Metric

2.1 Basics

• Consider the Linear Programming (LP) problem:
  $$\begin{array}{rl}{W}_{\mathbf{S},1}(\mathbb{P},\mathbb{Q})=\underset{\pi }{\min }& \sum _{i=1}^m\sum _{j=1}^n\pi (i,j)s(i,j)\\ \text{ s.t. }& \sum _{i=1}^m\pi (i,j)=q_j,j\in [[n]]\\ & \sum _{j=1}^n\pi (i,j)=p_i,i\in [[m]]\\ & \pi (i,j)\ge 0,\forall i,j\end{array}$$
  the objective value is the order-1 Wasserstein distance between distributions $\mathbb{P}$ and $\mathbb{Q}$
• Similarly, by defining a cost matrix ${\mathbf{S}}^t=\left((s(i,j){)}^t\right)$, t), we have the order-$t$ Wasserstein distance
  $$W_{\mathbf{S},t}(\mathbb{P},\mathbb{Q})={\left(W_{{\mathbf{S}}^t,1}(\mathbb{P},\mathbb{Q})\right)}^{1/t}$$
• The above LP formulation is equivalent to the well-known transportation problem (Bertsimas and Tsitsiklis, 1997)
  

2.2 A Distance Metric

• In this section we establish that the Wasserstein distance $W_{\mathbf{S},t}(\mathbb{P},\mathbb{Q})$ is a distance metric, assuming that the underlying cost $s(i,j)$ is a proper distance metric.
• $W_{\mathbf{S},1}(\mathbb{P},\mathbb{Q})$, viewed as a function of the vectors $p$ and $q$ corresponding to $\mathbb{P}$ and $\mathbb{Q}$, is a convex function.

2.3 The Dual Problem

• The dual of LP in section 2.1
  $$\begin{array}{rl}{W}_{\mathbf{S},1}(\mathbb{P},\mathbb{Q})=\underset{\mathbf{f},\mathbf{g}}{\max }& \sum _{i=1}^m{g}_i{p}_i+\sum _{j=1}^n{f}_j{q}_j\\ \text{ s.t. }& f_j+g_i\le s(i,j),i\in [[m]],j\in [[n]]\end{array}$$
• interpretation

2.3.1 Arbitrary Measures and Kantorovich Duality

• Primal:
  $$W_{s,1}(\mathbb{P},\mathbb{Q})=\underset{\pi }{\min }{\int }_{{\mathcal{Z}}_1\times {\mathcal{Z}}_2}s\left({\mathbf{z}}_1,{\mathbf{z}}_2\right)\mathrm{d}\pi \left({\mathbf{z}}_1,{\mathbf{z}}_2\right)$$
  $$W_{s,t}(\mathbb{P},\mathbb{Q})={\left(W_{s^t,1}(\mathbb{P},\mathbb{Q})\right)}^{1/t}$$
  $$s^t\left({\mathbf{z}}_1,{\mathbf{z}}_2\right)={\left(s\left({\mathbf{z}}_1,{\mathbf{z}}_2\right)\right)}^t$$
• Dual:
  $$\begin{array}{rl}{W}_{s,1}(\mathbb{P},\mathbb{Q})=\underset{f,g}{\sup }& {\int }_{{\mathcal{Z}}_1}g\left({\mathbf{z}}_1\right)d\mathbb{P}\left({\mathbf{z}}_1\right)+{\int }_{{\mathcal{Z}}_2}f\left({\mathbf{z}}_2\right)\mathrm{d}\mathbb{Q}\left({\mathbf{z}}_2\right)\\ \text{ s.t. }& f\left({\mathbf{z}}_2\right)+g\left({\mathbf{z}}_1\right)\le s\left({\mathbf{z}}_1,{\mathbf{z}}_2\right),{\mathbf{z}}_1\in {\mathcal{Z}}_1,{\mathbf{z}}_2\in {\mathcal{Z}}_2,\end{array}$$

2.4 Some Special Cases

2.5 The Transport Cost Function

• we discuss a number of different scenarios on what may be known regarding the data and the implied appropriate corresponding cost function
  • sparse
  • dense
  • group sparsity

2.5.1 Transport Cost Function via Metric Learning

2.6 Robustness of the Wasserstein Ambiguity Set

• Theorem 2.6.1. Suppose we are given two probability distributions $\mathbb{P}$ and ${\mathbb{P}}_{out}$, and the mixture distribution ${\mathbb{P}}_{mix}$ is a convex combination of the two: ${\mathbb{P}}_{mix}=q{\mathbb{P}}_{out}+(1-q)\mathbb{P}$. Then, for any cost function $s$,
  $$\frac{W_{s,1}\left({\mathbb{P}}_{\text{out }},{\mathbb{P}}_{\text{mix }}\right)}{W_{s,1}\left(\mathbb{P},{\mathbb{P}}_{\text{mix }}\right)}=\frac{1-q}{q}$$
• We claim that when $q$ is small, if the Wasserstein ball radius $ϵ$ is chosen judiciously, the true distribution $\mathbb{P}$ will be included in the $ϵ$-Wasserstein ball $\Omega$ while the outlying distribution ${\mathbb{P}}_{out}$ will be excluded.

2.7 Setting the Radius of the Wasserstein Ball

• In the next two subsections we discuss two practical radius selection approaches

2.7.1 Measure Concentration

2.7.2 Robust Wasserstein Profile Inference