paper_longformer

LongformerModel(
  (embeddings): LongformerEmbeddings(
    (word_embeddings): Embedding(30522, 768, padding_idx=1)
    (position_embeddings): Embedding(512, 768, padding_idx=1)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (encoder): LongformerEncoder(
    (layer): ModuleList(
      (0): LongformerLayer(
        (attention): LongformerAttention(
          (self): LongformerSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (query_global): Linear(in_features=768, out_features=768, bias=True)
            (key_global): Linear(in_features=768, out_features=768, bias=True)
            (value_global): Linear(in_features=768, out_features=768, bias=True)
          )
          (output): LongformerSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): LongformerIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): LongformerOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )

longformer
输入向量化层
中间encoding编码层，这里面有多头注意力层，有分局部注意力和全局注意力
输出层
中间线性层
输出层
把结构打印出来，一清二楚，非常明了

目录结构就在这里，其实路径和windows一样，只是前面的home可能会有区别而已，可以用~ 代替
cd ~/.cache/huggingface/transformers

studio或者jupyter或者colab直接把变量打印出即可，不必用print,默认是最后的变量，要打印出相应的size，直接张量.size()即可

一样都是统一继承LongformerPreTrainedModel这个模型，只是一个是标准的特征抽取没有接下游任务，其他的是接了下游任务针对特定任务的类而已。继承的这个类，涉及到参数的定义和加载等。