Adversarially Robust Generalization Requires More Data

Schmidt L, Santurkar S, Tsipras D, et al. Adversarially Robust Generalization Requires More Data[C]. neural information processing systems, 2018: 5014-5026.

@article{schmidt2018adversarially,
title={Adversarially Robust Generalization Requires More Data},
author={Schmidt, Ludwig and Santurkar, Shibani and Tsipras, Dimitris and Talwar, Kunal and Madry, Aleksander},
pages={5014–5026},
year={2018}}

概

本文在二分类高斯模型和伯努利模型上分析adversarial, 指出对抗稳定的模型需要更多的数据支撑.

主要内容

高斯模型定义: 令${\theta }^{\ast }\in {\mathbb{R}}^n$为均值向量, $\sigma >0$, 则$({\theta }^{\ast },\sigma )$-高斯模型按照如下方式定义: 首先从等概率采样标签 y ∈ { ± 1 } y \in \{\pm 1\} y∈{±1}, 再从$\mathcal{N}(y\cdot {\theta }^{\ast },{\sigma }^{2}I)$中采样$x\in {\mathbb{R}}^d$.

伯努利模型定义: 令 θ ∗ ∈ { ± 1 } d \theta^* \in \{\pm1\}^d θ∗∈{±1}d为均值向量, $\tau >0$, 则$({\theta }^{\ast },\tau )$-伯努利模型按照如下方式定义: 首先等概率采样标签 y ∈ { ± 1 } y \in \{\pm 1\} y∈{±1}, 在从如下分布中采样 x ∈ { ± 1 } d x \in \{\pm 1\}^d x∈{±1}d:
$$x_i=\{\begin{array}{rl}y\cdot {\theta }_i^{\ast }& \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}1/2+\tau \\ -y\cdot {\theta }_i^{\ast }& \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}1/2-\tau \end{array}$$

分类错误定义: 令 P : R d × { ± 1 } → R \mathcal{P}: \mathbb{R}^d \times \{\pm 1\} \rightarrow \mathbb{R} P:Rd×{±1}→R为一分布, 则分类器 f : R d → { ± 1 } f:\mathbb{R}^d \rightarrow \{\pm1\} f:Rd→{±1}的分类错误$\beta$定义为$\beta ={\mathbb{P}}_{(x,y)\sim \mathcal{P}}[f(x)\ne y]$.

Robust分类错误定义: 令 P : R d × { ± 1 } → R \mathcal{P}: \mathbb{R}^d \times \{\pm 1\} \rightarrow \mathbb{R} P:Rd×{±1}→R为一分布, $\mathcal{B}:{\mathbb{R}}^d\to \mathcal{P}({\mathbb{R}}^d)$为一摄动集合. 则分类器 f : R d → { ± 1 } f:\mathbb{R}^d \rightarrow \{\pm1\} f:Rd→{±1}的$\mathcal{B}$-robust 分类错误率$\beta$定义为$\beta ={\mathbb{P}}_{(x,y)\sim \mathcal{P}}[\exists x^{\prime }\in \mathcal{B}(x):f(x^{\prime })\ne y]$.

注: 以${\mathcal{B}}_p^{ϵ}(x)$表示 { x ′ ∈ R d ∣ ∥ x ′ − x ∥ p ≤ ϵ } \{x' \in \mathbb{R}^d|\|x'-x\|_p \le \epsilon\} {x′∈Rd∣∥x′−x∥p​≤ϵ}.

高斯模型

upper bound

定理18: 令 ( x 1 , y 1 ) , … , ( x n , y n ) ∈ R d × { ± 1 } (x_1,y_1),\ldots, (x_n,y_n) \in \mathbb{R}^d \times \{\pm 1\} (x1​,y1​),…,(xn​,yn​)∈Rd×{±1} 独立采样于同分布$({\theta }^{\ast },\sigma )$-高斯模型, 且$\Vert {\theta }^{\ast }{\Vert }_2=\sqrt{d}$. 令$\hat{w}:=\bar{z}/\Vert \bar{z}\Vert \in {\mathbb{R}}^d$, 其中$\bar{z}=\frac{1}{n}{\sum }_{i=1}^n{y}_i{x}_i$. 则至少有$1-2\exp (-\frac{d}{8({\sigma }^2+1)})$的概率, 线性分类器$f_{\hat{w}}$的分类错误率至多为:
$$\exp (-\frac{(2\sqrt{n}-1{)}^{2}d}{2(2\sqrt{n}+4\sigma {)}^2{\sigma }^2}).$$

定理21: 令 ( x 1 , y 1 ) , … , ( x n , y n ) ∈ R d × { ± 1 } (x_1,y_1),\ldots, (x_n,y_n) \in \mathbb{R}^d \times \{\pm 1\} (x1​,y1​),…,(xn​,yn​)∈Rd×{±1} 独立采样于同分布$({\theta }^{\ast },\sigma )$-高斯模型, 且$\Vert {\theta }^{\ast }{\Vert }_2=\sqrt{d}$. 令$\hat{w}:=\bar{z}/\Vert \bar{z}\Vert \in {\mathbb{R}}^d$, 其中$\bar{z}=\frac{1}{n}{\sum }_{i=1}^n{y}_i{x}_i$. 如果
$$ϵ\le \frac{2\sqrt{n}-1}{2\sqrt{n}+4\sigma }-\frac{\sigma \sqrt{2\log 1/\beta }}{\sqrt{d}},$$

则至少有$1-2\exp (-\frac{d}{8({\sigma }^2+1)})$的概率, 线性分类器$f_{\hat{w}}$的${\ell }_{\infty }^{ϵ}$-robust 分类错误率至多为$\beta$.

lower bound

定理11: 令$g_n$为任意的学习算法, 并且, $\sigma >0,ϵ\ge 0$, 设$\theta \in {\mathbb{R}}^d$从$\mathcal{N}(0,I)$中采样. 并从$(\theta ,\sigma )$-高斯模型中采样$n$个样本, 由此可得到分类器 f n : R d → { ± 1 } f_n: \mathbb{R}^d \rightarrow \{\pm 1\} fn​:Rd→{±1}. 则分类器关于$\theta ,(y_1,\dots ,y_n),(x_1,\dots ,x_n)$的${\ell }_{\infty }^{ϵ}$-robust 分类错误率至少为
$$\frac{1}{2}{\mathbb{P}}_{v\sim \mathcal{N}(0,I)}[\sqrt{\frac{n}{{\sigma }^2+n}}\Vert v{\Vert }_{\infty }\le ϵ].$$

伯努利模型

upper bound

令 ( x , y ) ∈ R d × { ± 1 } (x, y) \in \mathbb{R}^d \times \{\pm1\} (x,y)∈Rd×{±1}从一$({\theta }^{\ast },\tau )$-伯努利模型中采样得到. 令$\hat{w}=z/\Vert z{\Vert }_2$, 其中$z=yx$. 则至少有$1-\exp (-\frac{{\tau }^{2}d}{2})$的概率, 线性分类器$f_{\hat{w}}$的分类错误率至多为$\exp (-2{\tau }^{4}d)$.

lower bound

引理30： 令 θ ∗ ∈ { ± 1 } d \theta^* \in \{\pm1\}^d θ∗∈{±1}d 并且关于$({\theta }^{\ast },\tau )-伯努利模型$考虑线性分类器$f_{{\theta }^{\ast }}$,
${\ell }_{\infty }^{\tau }$-robustness: $f_{{\theta }^{\ast }}$的${\ell }_{\infty }^{\tau }$-robust分类误差率至多为$2\exp (-{\tau }^{2}d/2)$.
${\ell }_{\infty }^{3\tau }$-nonrobustness: $f_{{\theta }^{\ast }}$的${\ell }_{\infty }^{3\tau }$-robust分类误差率至少为$1-2\exp (-{\tau }^{2}d/2)$.
Near-optimality of ${\theta }^{\ast }$: 对于任意的线性分类器, ${\ell }_{\infty }^{3\tau }$-robust 分类误差率至少为$\frac{1}{6}$.

定理31: 令$g_n$为任一线性分类器学习算法. 假设${\theta }^{\ast }$均匀采样自 { ± 1 } d \{\pm1\}^d {±1}d, 并从$({\theta }^{\ast },\tau )$-伯努利分布($\tau \le 1/4$)中采样$n$个样本, 并借由$g_n$得到线性分类器$f_w$.同时$ϵ<3\tau$且$0<\gamma <1/2$, 则当
$$n\le \frac{{ϵ}^2{\gamma }^2}{5000\cdot {\tau }^4\log (4d/\gamma )},$$
$f_w$关于${\theta }^{\ast },(y_1,\dots ,y_n),(x_1,\dots ,x_n)$的期望${\ell }_{\infty }^{ϵ}$-robust 分类误差至少为$\frac{1}{2}-\gamma$.