今日arXiv精选 | 近期必读的5篇Transformers相关论文

 关于 #今日arXiv精选 

这是「AI 学术前沿」旗下的一档栏目，编辑将每日从arXiv中精选高质量论文，推送给读者。

Fastformer: Additive Attention is All You Need

Category: NLP

Link: https://arxiv.org/abs/2108.09084

Abstract

Transformer is a powerful model for text understanding. It is inefficient due to its quadratic complexity to input sequence length. In Fastformer, instead of modeling the pair-wise interactionsbetween tokens, we first use additive attention mechanism to model global contexts.

Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text  Models

Category: NLP

Link: https://arxiv.org/abs/2108.08877

Abstract

Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast assequence-to-sequence mapping problems, it is unclear how to produce sentences from encoder-decoder models. We investigate three methods for extracting T5 sentences.

Do Vision Transformers See Like Convolutional Neural Networks?

Category: Computer Vision

Link: https://arxiv.org/abs/2108.0881

Abstract

Recent work has shown that (Vision) Transformer models (ViT) can achieve superior performance on image classification tasks. Are they acting like convolutional networks, or learning entirely different visual representations?

Is it Time to Replace CNNs with Transformers for Medical Images?

Category: Computer Vision

Link: https://arxiv.org/abs/2108.09038

Abstract

Vision transformers (ViTs) have appeared as a competitive alternative to CNNs. ViTs possess several interestingproperties that could prove beneficial for medical imaging tasks. While CNNs perform better when trained from scratch, ViTs are on par with CNNs when pretrained on ImageNet.

Video Relation Detection via Tracklet based Visual Transformer

Category: Computer Vision

Link: https://arxiv.org/abs/2108.08669

Abstract

Video Visual Relation Detection (VidVRD), has received significant attention over recent years. We apply the state-of-the-art video object tracklet detection pipeline MEGA and deepSORT. Then we perform VidVRD in a tracklet-based manner.

·