Conformal Prediction

Conformal Prediction in Classification

Conformal Coverage Guarantee

Given the calibration data set { ( X i , Y i ∗ ) } i = 1 n {\left \{\left ( X_i, {Y}_{i}^{*}\right ) \right \}}_{i=1}^{n} {(Xi​,Yi∗​)}i=1n​ and pretrained model $\hat{f}\left(\cdot \right)$ ($\hat{f}\left(X_i\right)\in {\left[0,1\right]}^{\left(K\right)}$).
The probability (or confidence) assigned to the true label is ${\hat{f}\left(X_i\right)}_{Y_i^{\ast }}$.

Calculate and sort the conformal scores: $s_i=s\left(X_i,Y_i^{\ast }\right)=1-{\hat{f}\left(X_i\right)}_{Y_i^{\ast }}$ ( { s 1 ≤ ⋯ ≤ s n } \left \{s_1 \leq \cdots \leq s_n \right \} {s1​≤⋯≤sn​}).

Obtain the $\frac{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }{n}$ quantile of { s i } i = 1 n {\left \{ s_i\right \}}_{i=1}^{n} {si​}i=1n​: q ^ = inf ⁡ { q : ∣ { i : s i ≤ q } ∣ n ≥ ⌈ ( n + 1 ) ( 1 − α ) ⌉ n } = s ⌈ ( n + 1 ) ( 1 − α ) ⌉ \hat{q}=\inf \left \{ q:\frac{\left | \left \{ i:s_i \leq q\right \}\right |}{n} \geq \frac{\left \lceil \left ( n+1\right )\left ( 1-\alpha \right )\right \rceil}{n} \right \} = {s}_{\left \lceil \left ( n+1\right )\left ( 1-\alpha \right )\right \rceil} q^​=inf{q:n∣{i:si​≤q}∣​≥n⌈(n+1)(1−α)⌉​}=s⌈(n+1)(1−α)⌉​.

Construct the prediction set of $\left(X_{test},Y_{test}^{\ast }\right)$: C ( X t e s t ) = { y : f ^ ( X t e s t ) y ≥ 1 − q ^ } = { y : s ( X t e s t , y ) ≤ q ^ } \mathcal{C}\left ( {X}_{test}\right )=\left \{ y: {\hat{f}\left ( {X}_{test}\right )}_{y} \geq 1-\hat{q} \right \}=\left \{ y: s\left ( {X}_{test}, y\right )\leq \hat{q}\right \} C(Xtest​)={y:f^​(Xtest​)y​≥1−q^​}={y:s(Xtest​,y)≤q^​}.

The event { Y t e s t ∗ ∈ C ( X t e s t ) } \left \{ {Y}_{test}^{*} \in \mathcal{C}\left ( {X}_{test}\right ) \right \} {Ytest∗​∈C(Xtest​)} is equivalent to { s ( X t e s t , Y t e s t ∗ ) ≤ q ^ } \left \{ s\left ( {X}_{test}, {Y}_{test}^{*}\right )\leq \hat{q}\right \} {s(Xtest​,Ytest∗​)≤q^​}.

By the exchangeability of $\left(X_1,Y_1\right),\cdots ,\left(X_n,Y_n\right),\left(X_{test},Y_{test}^{\ast }\right)$, we have $\mathcal{P}\left(s_{test}\le s_i\right)=\frac{i}{n+1}$.

Then we get the probability of conformal coverage: $\mathcal{P}\left(Y_{test}^{\ast }\in \mathcal{C}\left(X_{test}\right)\right)=\mathcal{P}\left(s_{test}\le \hat{q}\right)=\frac{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }{n+1}$.
The lower bound is $1-\alpha$, and the upper bound is $1-\alpha +\frac{1}{n+1}$.

Classification with Adaptive Prediction Set

Given { π k ( X i ) } k = 1 K {\left \{ {\pi}_{k}\left ( X_i\right )\right \}}_{k=1}^{K} {πk​(Xi​)}k=1K​ as the permutation of { k } k = 1 K {\left \{ k\right \}}_{k=1}^{K} {k}k=1K​ that sorts $\hat{f}\left(X_i\right)$ ($\left\{\widehat{{f\left(X_i\right)}_{{\pi }_1\left(X_i\right)}}\ge \cdots \ge \widehat{{f\left(X_i\right)}_{{\pi }_K\left(X_i\right)}}\right\}$).

Calculate the conformal scores: $s_i=s\left(X_i,Y_i^{\ast }\right)={\sum }_{j=1}^k{\hat{f}\left(X_i\right)}_{{\pi }_j\left(X_i\right)}$, where ${\pi }_k\left(X_i\right)=Y_i^{\ast }$.

Obtain the $\frac{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }{n}$ quantile of { s i } i = 1 n {\left \{ s_i\right \}}_{i=1}^{n} {si​}i=1n​: $\hat{q}$.

Prediction set: C ( X t e s t ) = { π 1 ( X t e s t ) , ⋯   , π k ( X t e s t ) } \mathcal{C}\left ( {X}_{test}\right )=\left \{ {\pi}_{1}\left ( {X}_{test} \right ), \cdots , {\pi}_{k}\left ( {X}_{test} \right )\right \} C(Xtest​)={π1​(Xtest​),⋯,πk​(Xtest​)}, where $k=\sup \left\{k^{\prime }:{\sum }_{j=1}^{k^{\prime }}{\hat{f}\left(X_{test}\right)}_{{\pi }_j\left(X_{test}\right)}<\hat{q}\right\}+1$.

(proof of bound not completed)

LLMs with Conformal Factualiy Guarantees

Given an input $X_{test}\in \mathcal{X}$, we get an output $\mathcal{L}\left(X_{test}\right)\in \mathcal{Y}$. The goal is $\mathcal{P}\left ( \mathcal{L}\left ( {X}_{test}\right ) ; is ; correct \right ) \geq 1-\alpha $.

The correctness of $\mathcal{L}\left(X_{test}\right)$ is equivalent to the entailment relation $Y_{test}^{\ast }\Rightarrow \mathcal{L}\left(X_{test}\right)$.

Define the entailment set of $\mathcal{L}\left ( {X}_{test}\right ) $: E ( L ( X t e s t ) ) = { y ∈ Y : y ⇒ L ( X t e s t ) } \mathcal{E}\left ( \mathcal{L}\left ( {X}_{test}\right ) \right ) = \left \{ y\in\mathcal{Y}:y\Rightarrow \mathcal{L}\left ( {X}_{test}\right ) \right \} E(L(Xtest​))={y∈Y:y⇒L(Xtest​)}, then { Y t e s t ∗ ⇒ L ( X t e s t ) } \left\{ {Y}_{test}^{*}\Rightarrow \mathcal{L}\left ( {X}_{test}\right )\right\} {Ytest∗​⇒L(Xtest​)} is equivalent to { Y t e s t ∗ ∈ E ( L ( X t e s t ) ) } \left\{{Y}_{test}^{*} \in \mathcal{E}\left ( \mathcal{L}\left ( {X}_{test}\right ) \right )\right\} {Ytest∗​∈E(L(Xtest​))}.

Construct { F t ( X i ) } t ∈ T {\left \{ {\mathcal{F}}_{t}\left ( X_i\right )\right \}}_{t\in \mathcal{T}} {Ft​(Xi​)}t∈T​ following the \textit{nested property} (i.e., $\forall t_1,t_2\in \mathcal{T},t_1\le t_2\to {\mathcal{F}}_{t_1}\subseteq {\mathcal{F}}_{t_2}$), where ${\mathcal{F}}_t\left(X_i\right)$ is the entailment set of $ {F}{t}\left ( X_i, \mathcal{L}\left ( X_i\right )\right )$ (i.e., ${\mathcal{F}}_t\left(X_i\right)=\mathcal{E}\left(F_t\left(X_i,\mathcal{L}\left(X_i\right)\right)\right)$) and $F_t\left(X_i,\mathcal{L}\left(X_i\right)\right)$ is the output calibrated by the ``back off’’ function $F_t(\cdot )$ with the safe threshold $t$ from the base output $\mathcal{L}\left(X_i\right)$ (by removing unreliable sub-claims). (${F}{\sup \mathcal{T}}\left ( X_i, \mathcal{L}\left ( X_i\right )\right )=\varnothing $, $F_0\left(X_i,\mathcal{L}\left(X_i\right)\right)=\mathcal{L}\left(X_i\right)$)

Define the conformal scores: r ( X i , Y i ∗ ) = inf ⁡ { t : ∀ j ≥ t , Y i ∗ ∈ F j ( X i ) } = inf ⁡ { t : ∀ j ≥ t , Y i ∗ ⇒ F j ( X i , L ( X i ) ) } r\left (X_i,{Y}_{i}^{*} \right )=\inf \left \{t:\forall j\geq t,{Y}_{i}^{*} \in\mathcal{F}_j\left ( X_i\right ) \right \}=\inf \left \{t:\forall j\geq t,{Y}_{i}^{*} \Rightarrow F_j\left (X_i,\mathcal{L}\left ( X_i \right ) \right ) \right \} r(Xi​,Yi∗​)=inf{t:∀j≥t,Yi∗​∈Fj​(Xi​)}=inf{t:∀j≥t,Yi∗​⇒Fj​(Xi​,L(Xi​))}
(min safe threshold that holds the true label).

Obtain the $\frac{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }{n}$ quantile of { r ( X i , Y i ∗ ) } i = 1 n {\left \{ r\left (X_i,{Y}_{i}^{*} \right )\right \}}_{i=1}^{n} {r(Xi​,Yi∗​)}i=1n​: $\hat{q}=r_{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }$ (sort $r_1\le \cdots \le r_n$).

{ r t e s t ≤ q ^ } \left \{ {r}_{test} \leq \hat{q} \right \} {rtest​≤q^​} is equivalent to { Y t e s t ∗ ⇒ F q ^ ( X t e s t , L ( X t e s t ) ) } \left \{ {Y}_{test}^{*}\Rightarrow {F}_{\hat{q}}\left ( {X}_{test}, \mathcal{L}\left ( {X}_{test}\right )\right )\right \} {Ytest∗​⇒Fq^​​(Xtest​,L(Xtest​))} (i.e., $\hat{q}$ is a safe threshold).

Then $\mathcal{P}\left(F_{\hat{q}}\left(X_{test},\mathcal{L}\left(X_{test}\right)\right)iscorrect\right)=\mathcal{P}\left(r_{test}\le \hat{q}\right)=\frac{\lceil \left(n+1\right)\left(1-\alpha \right)\rceil }{n+1}\in \left[1-\alpha ,1-\alpha +\frac{1}{n+1}\right]$.

We can evaluate the entailment of the current output controlled by $t$ by only evaluating the sub-claims of the base output once and computing the \textit{supremum} (safe threshold) over the sub-claims within the current output.

For sub-claims { c m } m = 1 M i t {\left \{ c_m\right \}}_{m=1}^{{M}_{it}} {cm​}m=1Mit​​ and $Y_i^{\ast }\in \mathcal{Y}$, { Y i ∗ ⇒ F t ( X i , L ( X i ) ) } ⇔ { ∀ m ∈ M i t , Y i ∗ ⇒ c m } \left \{ {Y}_{i}^{*}\Rightarrow F_t\left ( X_i,\mathcal{L}\left ( X_i\right )\right ) \right \}\Leftrightarrow \left \{\forall m \in {M}_{it}, {Y}_{i}^{*}\Rightarrow c_m \right \} {Yi∗​⇒Ft​(Xi​,L(Xi​))}⇔{∀m∈Mit​,Yi∗​⇒cm​}, where $M_{it}$ is the current number of the sub-claims within the $i$-th output controlled (or accepted) by $t$.

Then the conformal scores are: r ( X i , Y i ∗ ) = inf ⁡ { t : ∀ j ≥ t , ∀ c ∈ F j ( X i , L ( X i ) ) , Y i ∗ ⇒ c } r\left (X_i,{Y}_{i}^{*} \right )=\inf \left \{t:\forall j\geq t, \forall c\in F_j\left (X_i,\mathcal{L}\left ( X_i\right ) \right ),{Y}_{i}^{*} \Rightarrow c \right \} r(Xi​,Yi∗​)=inf{t:∀j≥t,∀c∈Fj​(Xi​,L(Xi​)),Yi∗​⇒c}.

As for the entailment set, $\mathcal{E}\left(F_t\left(X_i,\mathcal{L}\left(X_i\right)\right)\right)={\bigcap }_m^{M_{it}}\mathcal{E}\left(c_m\right)$, and then the conformal scores are $r\left(X_i,Y_i^{\ast }\right)=\inf \left\{t:\forall j\ge t,Y_i^{\ast }\in {\bigcap }_m^{M_{ij}}\mathcal{E}\left(c_m\right)\right\}$.

Instead of guaranteeing full factuality, we want $a\in \left[0,1\right]$ fraction of the accepted sub-claims (i.e., $F_t\left(X_i,\mathcal{L}\left(X_i\right)\right)$) to be factual (Partial entailment keeps the min safe threshold small, which mitigates the issue of $\hat{q}$ being so large that the accepted sub-claims are uninformative or even empty).

Then the conformal scores with acceptable entailment level $a\in \left[0,1\right]$ are:\ $r_a\left(X_i,Y_i^{\ast }\right)=\inf \left\{t\in \mathcal{T}:\forall j\ge t,{\mathcal{M}}_{Y_i^{\ast }}\left(F_j\left(X_i,\mathcal{L}\left(X_i\right)\right)\right)\ge a\right\}$, where ${\mathcal{M}}_{Y_i^{\ast }}\left(F_j\left(X_i,\mathcal{L}\left(X_i\right)\right)\right)=\frac{1}{M_{ij}}{\sum }_m^{M_{ij}}{\text{1}}_{Y_i^{\ast }\Rightarrow c_m}$, and $M_{ij}$ is the current number of the $i$-th output controlled by threshold $j$.

The event { r a ( X t e s t , Y t e s t ∗ ) ≤ q ^ } \left \{ r_a\left ( {X}_{test}, {Y}_{test}^{*} \right ) \leq \hat{q}\right \} {ra​(Xtest​,Ytest∗​)≤q^​} implies $\left\{{\mathcal{M}}_{Y_{test}^{\ast }}\left(F_{\hat{q}}\left(X_{test},\mathcal{L}\left(X_{test}\right)\right)\right)\ge a\right\}$ (not equivalent).