[iBOT] Image BERT Pre-Training with Online Tokenizer

1、目的

        探索visual tokenizer编码下的MIM（Masked Image Modeling）

2、方法

        iBOT（image BERT pre-training with Online Tokenizer）

        1）knowledge distillation（KD）

                distill knowledge from the tokenizer

        2）self-distillation

                twin teacher as online tokenizer

        3）visual tokenizer

                -> transform the masked patches to supervisory signals for the target model

                -> 通过enforce the similarity of cross-view images on class tokens，来捕获到high-level visual semantics 

                -> 无须额外的训练，通过momentum update来和MIM一同被优化

                -> online，而不是pre-fixed

        4）网络结构

                -> ，

                -> 用softmax之后的token，而非ont-hot

        5）multi-crop

        6）MIM

                                        ​​​​​​​        

3、结论

        1）iBOT is more scalable to larger models

        2）iBOT requires more data to train larger model