Overview Of MNAR Matrix Completion Under Nuclear Norm assumption

Why Use Propensity Score

Setup

Set up three matrices:

1. signol matrix $S\in {\mathbb{R}}^{m,n}$.
2. noise matrix $W\in {\mathbb{R}}^{m,n}$.
3. probability matrix $P\in [0,1{]}^{m,n}$ independent of everything else

• Suppose we observe a matrix $X^{\ast }\in {\mathbb{R}}^{m,n}$ without missing entries
  $$X^{\ast }:=S+W$$
  • In order to estimate $S$ with low MSE, $\hat{S}$ can be obtained from $$L_{\text{full MSE}}(\hat{S}):=\frac{1}{mn}\sum _{u=1}^m\sum _{i=1}^n({\hat{S}}_{u,i}-X_{u,i}^{\ast })\text{\hspace{1em}(1)}$$
• However, we only observe a matrix X ∈ ( R ∪ { ⋆ } ) m × n X \in (\mathbb{R} \cup \{ \star \} )^{m \times n} X∈(R∪{⋆})m×n, in which each entry is a revealed rating $r$ or a unrevealed rating $\star$.
  $$X_{u,i}=S_{u,i}+W_{u,i}\text{ with probability }{P}_{u,i}$$
  • In order to estimate $S$ with low MSE, $\hat{S}$, we need $$L_{MSE}(\hat{S}):=\frac{1}{|\Omega |}\sum _{(u,i)\in \Omega }({\hat{S}}_{u,i}-X_{u,i}{)}^2\text{\hspace{1em}(2)}$$
    If the probability of every entry in X being revealed is the same, $P_{u,i}=C$ for all u and I, then (2) is un unbiased estimate of (1).
    However, the above assumption won’t be true in reality because the missing/revealed probability would be different for different u, i. Therefore, we need use inverse propensity score/inverse probability to solve the issue.

Conditional On Propensity Scores

• Inverse Propensity Score Technique: $$L_{IPS-MSE}(\hat{S}|P):=\frac{1}{mn}\sum _{(u,i)\in \Omega }\frac{({\hat{S}}_{u,i}-X_{u,i}{)}^2}{P_{u,i}}\text{\hspace{1em}(3)}$$, which is an unbiased estimate of (1)
• Appendix: Suppose $P$ is known, then estimate $L_{IPS-MSE}$ in (3) is an unbiased estimate for $L_{\text{full MSE}}(\hat{S})$ in (1) is unbiased estimator or (1): L I P S − M S E ( S ^ ∣ P ) : = 1 m n ∑ u = 1 m ∑ i = 1 n 1 { ( u , i ) ∈ Ω } ( S ^ u , i − X u , i ∗ ) 2 P u , i L_{IPS-MSE}(\hat{S}|P) := \frac{1}{mn} \sum^m_{u=1} \sum^n_{i=1} 1\{(u, i) \in \Omega \} \frac{(\hat{S}_{u,i}-X^*_{u,i})^2}{P_{u,i}} LIPS−MSE​(S^∣P):=mn1​u=1∑m​i=1∑n​1{(u,i)∈Ω}Pu,i​(S^u,i​−Xu,i∗​)2​, the we take expectation of it with respect to which entries are revealed: $$E_{\Omega }[L_{\text{full MSE}}(\hat{S})]=\frac{1}{mn}\sum _{u=1}^m\sum _{i=1}^n{P}_{u,i}\frac{({\hat{S}}_{u,i}-X_{u,i}^{\ast }{)}^2}{P_{u,i}}=L_{\text{full MSE}}(\hat{S})$$

Nulear Norm Regularization

• Overall, we add nuclear norm regularization to (3) and the final objective function is the following: $$\hat{S}=\underset{\Gamma \in {\mathbb{R}}^{m\times n}}{\mathrm{argmin}}{L}_{\text{IPS-MSE}}(\Gamma |P)+\lambda ||\Gamma |{|}_{\ast }\text{\hspace{1em}(4**)}$$, where $\lambda >0$ is user-specified parameter, and $||\cdot |{|}_{\ast }$ denotes the nuclear norm, $||A|{|}_{\ast }=trace(\sqrt{A^{T}A})={\sum }_{i=1}^{\text{min of (m, n)}}{\sigma }_i(A)$

Estimate Propensity Score

We only talk about one bit matrix completion of the propensity score estimation in this article

1bit Matrix Completion

Setup:

1. $M$: Missingness Mask Matrix used to estimate $P$
   * M ∈ { 0 , 1 } m × n \in \{0,1 \}^{m \times n} ∈{0,1}m×n
   * M u , i = 1 { X u , i ≠ ⋆ } M_{u,i}=1\{X_{u,i} \ne \star \} Mu,i​=1{Xu,i​​=⋆}
2. $A\in {\mathbb{R}}^{m\times n}$: parameter matrix
   * A ∈ F τ , γ : = { Γ ∈ R m × n : ∣ ∣ Γ ∣ ∣ ∗ ≤ τ m n , ∣ ∣ Γ ∣ ∣ max ⁡ ≤ γ } A\in \mathcal{F}_{\tau, \gamma} := \{ \Gamma \in \mathbb{R}^{m \times n} : ||\Gamma||_* \le \tau \sqrt{mn}, ||\Gamma||_{\max} \le \gamma \} A∈Fτ,γ​:={Γ∈Rm×n:∣∣Γ∣∣∗​≤τmn ​,∣∣Γ∣∣max​≤γ}, $\tau >0$ and $\gamma >0$ are user-specified parameters.
3. $P=\sigma (A_{u,i})$, $\sigma$ can, for instance, be logistic function $\sigma (x)=\frac{1}{(1+e^{-x})}$

1BitMC is given as follows:
4. Solve the constrained Bernoulli maximum likelihood problem:
$$\hat{A}=\underset{\Gamma \in {\mathcal{F}}_{\tau ,\gamma }}{\mathrm{argmax}}\sum _{u=1}^m\sum _{i=1}^n[M_{u,i}\log \sigma ({\Gamma }_{u,i})+(1-M_{u,i})log(1-\sigma ({\Gamma }_{u,i}))]$$
5. Construct the matrix $\hat{P}\in [0,1{]}^{m\times n}$

Take away meesage for 1bitmc

The algorithm applies one bit matrix compeltion on mask matrix to find a reconstructed matrix penalized with nuclear norm that can achieve the maximum of bernouli likelihood.

Naive Bayes （Tobias Schnabel ）

Let’s look at the formula of Naive Bayes:
$$\begin{array}{rl}\hat{y}& =\underset{y\in \mathfrak{Y}}{\mathrm{argmax}}p(y|\text{x};\mathbf{\theta })\\ & =\underset{y\in \mathfrak{Y}}{\mathrm{argmax}}p(\text{x}|y;\mathbf{\theta })p(y)\end{array}$$

• $p(\text{x}|y;\mathbf{\theta })$ can be compare to a model of tossing a dice with V faces for n times which is proportional to multinomial distribution

$$\text{likelihood=}p(\text{x}|y;\mathbf{\theta })\propto {\Pi }_{i=1}^V{\theta }_{i,y}^{x_i}$$

• Prior
  $$p(y)={\theta }_y$$

• Number of parameters in ; θ = { θ y , θ i , y } , ∀ i , y ;\boldsymbol \theta = \{\theta_y, \theta_{i, y}\}, \forall i, y ;θ={θy​,θi,y​},∀i,y:
  $$K+KV=(V+1)K$$

We need first to know some assumptions used in propensity score wiki

• Strongly Ignorable: If the potential outcomes are independent of teratment conditional on background variables X. $$r_0,r_1\perp Z|X$$
• Balancing score: $b(x)$ is a function of the observed covariates $X$ such that the conditional distribution of X given $b(X)$ is the same for treated $(Z=1)$ and control $(Z=0)$: $$Z\perp X|b(x)$$, the most trivial function is b(X)=X or e(X)=propensity score. (My thoughts: we can think this of a confounder relation in three nodes graph such that X is the confounder affect both treatment Z and response Y. Once we condiction on b(X), the confounder effect of X on Z can be removed)
• Propensity Score: is the probability of a unit being assinged to a particular treatment given a set of observed covariates. $$e(x)\stackrel{def}{=}\mathrm{Pr}(Z=1|X=x)$$
• Main Thereoms:
  • The propensity score $e(x)$ is a balancing score
  • The propensity score is the coarsest balancing score function because it takes a (possibly) multidimensional object ($X_i$) to be transofrmed into one dimension, while b(X)=X is the finnest one
  • If treatment assignment is strongly ignorable given X then:
    • it is also strongly ignorable given any balancing function. Specifically, given the propensity score: $$(r_0,r_1\perp Z|e(X))$$

Now, we will look at the Naive Bayes approach proposed by Schnabel.
$$P(O_{u,i}=1|Y_{u,i}=r)=\frac{P(Y=r|O=1)P(O=1)}{P(Y=r)}$$

• The subscripts $u,i$ are dropped. This means that Parameters are tied across all $u$ and $i$
• $P(Y=r|O=1)$ and $P(O=1)$: can be obtained by counting observed ratings in Missing Not At Random (MNAR) data
• $P(Y=r)$: we need a sample of Missing Completely At Random (MCAR) data to be combined with the existing Missing Not At Random.

Take away meesage for Naive Bayes

1. This algorithm first have a utilize missing completely at random once to find the normalized term in denominator. Second, we should think the likelihood part $P(Y=r|O=1)$ in naive bayes as tossing a dice model where each side is each rating scaled r from 1 to 5. So the estimated probability will be same for the same ratings rather than different probabilities for different pairs of user and item.

2. In propensity assumption senario, the naive bayes approach seems not make sense. The strongly ignorablity says $(Y_0,Y_1\perp O|e(X))$. However, naive approach is $e(X)=P(O|Y)$ which is little bit confusing and does not make sense in fact.
Reference
MNAR_1BitMC(Ma)
Naive_Bayes(Tobias Schnabel)