我的NLP

环境安装

我在重新安装这个环境，总是遇到transformers的包的问题，

建议下次装环境时，保持网络通畅

而且创建环境时，不要声明python版本

下面是我创建的环境

name: pytorch-nlp
channels:
  - pytorch
  - defaults
dependencies:
  - blas=1.0=mkl
  - ca-certificates=2022.3.29=haa95532_1
  - certifi=2021.10.8=py38haa95532_2
  - cudatoolkit=10.2.89=h74a9793_1
  - freetype=2.10.4=hd328e21_0
  - intel-openmp=2021.4.0=haa95532_3556
  - jpeg=9b=hb83a4c4_2
  - libpng=1.6.37=h2a8f88b_0
  - libtiff=4.2.0=hd0e1b90_0
  - libuv=1.40.0=he774522_0
  - libwebp=1.2.2=h2bbff1b_0
  - lz4-c=1.9.3=h2bbff1b_1
  - mkl=2021.4.0=haa95532_640
  - mkl-service=2.4.0=py38h2bbff1b_0
  - mkl_fft=1.3.1=py38h277e83a_0
  - mkl_random=1.2.2=py38hf11a4ad_0
  - ninja=1.10.2=py38h559b2a2_3
  - numpy=1.21.5=py38h7a0a035_1
  - numpy-base=1.21.5=py38hca35cd5_1
  - openssl=1.1.1n=h2bbff1b_0
  - pillow=9.0.1=py38hdc2b20a_0
  - pip=21.2.2=py38haa95532_0
  - python=3.8.13=h6244533_0
  - pytorch=1.8.0=py3.8_cuda10.2_cudnn7_0
  - setuptools=61.2.0=py38haa95532_0
  - six=1.16.0=pyhd3eb1b0_1
  - sqlite=3.38.2=h2bbff1b_0
  - tk=8.6.11=h2bbff1b_0
  - torchaudio=0.8.0=py38
  - torchvision=0.9.0=py38_cu102
  - typing_extensions=4.1.1=pyh06a4308_0
  - vc=14.2=h21ff451_1
  - vs2015_runtime=14.27.29016=h5e58377_2
  - wheel=0.37.1=pyhd3eb1b0_0
  - wincertstore=0.2=py38haa95532_2
  - xz=5.2.5=h62dcd97_0
  - zlib=1.2.12=h8cc25b3_2
  - zstd=1.4.9=h19a0ad4_0
  - pip:
    - charset-normalizer==2.0.12
    - click==8.1.2
    - colorama==0.4.4
    - filelock==3.6.0
    - huggingface-hub==0.5.1
    - idna==3.3
    - joblib==1.1.0
    - packaging==21.3
    - pyparsing==3.0.8
    - pyyaml==6.0
    - regex==2022.3.15
    - requests==2.27.1
    - sacremoses==0.0.49
    - tokenizers==0.12.1
    - tqdm==4.64.0
    - transformers==4.19.0.dev0
    - urllib3==1.26.9
prefix: C:\Software\anaconda\envs\pytorch-nlp

我创建这么多环境，只有两个操作，算三个吧

conda create -n pytorch-nlp #没有声明python版本

conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch

cd C:\Software\anaconda\Lib\site-packages\transformers-main
python setup.py install

附：如何导出环境

conda env export > env.txt

安装Huggingface

https://github.com/huggingface/transformers

到这个网站下载下来，放到

C:\Software\anaconda\Lib\site-packages

这里，我喜欢放在这里，之后，在新建的conda环境里cd到这里，输入

python setup.py install

即可安装

还有HuggingFace的其它Datasets包，Tokenizers包，也可以这么安装

参考教程

最重要的教程

Introduction - Hugging Face Course

其余是一些同人

【Huggingface Transformers】保姆级使用教程—上 - 知乎

【Huggingface Transformers】保姆级使用教程02—微调预训练模型 Fine-tuning - 知乎

HuggingFace 重要链接

https://huggingface.co/models

遇到translation pipeline，可能出现如下错误:
 

ValueError: This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.

说明需要安装sentencepiece

 conda install sentencepiece

Hugging Face Pipeline

我下载了模型，在E盘，模型文件可以一直保留

zero-shot-classification

from transformers import pipeline
path = r"E:\HuggingFace\transformers\facebook\bart-large-mnli"
clssifier = pipeline("zero-shot-classification",model=path)
print(clssifier(
    "This is a course about the Transformers library!",
    candidate_labels = ["education","politics","business"],
))

text-generation

from transformers import pipeline
path = r"E:\HuggingFace\transformers\distilgpt2"
generator = pipeline("text-generation",model=path)
print(generator(
    "In this course,we will teach you how to",
    max_length = 30,
    num_return_sequences = 2,
))

fill-mask

from transformers import pipeline
path = r"E:\HuggingFace\transformers\bert-base-uncased"
unmasker = pipeline("fill-mask",model=path)
print(unmasker("This course will teach you all about [MASK] models.",top_k = 2))

ner

from transformers import pipeline
path = r"E:\HuggingFace\transformers\dslimbert\large-NER"
ner = pipeline("ner",grouped_entities = True,model=path)
print(ner("My name is Sylvain and I work at Hugging Face in Brooklyn."))

question-answering

from transformers import pipeline
path = r"E:\HuggingFace\transformers\bert-large-uncased-whole-word-masking-finetuned-squad"
question_answerer = pipeline("question-answering",model=path)
print(question_answerer(question="Where do I work?",
                        context="My name is Sylvain and I work at Hugging Face in Brooklyn."
                        )
      )

summarization

from transformers import pipeline
path = r"E:\HuggingFace\transformers\google\pegasus-xsum"
summarizer = pipeline("summarization",model=path)
print(summarizer(
    # 夜に駆ける
        """
    Like sinking, like melting
    
    At night when the sky of only two people spreads
    
    It was just "goodbye"
    
    I understood everything in that one word
    
    The sky where the sun has set and your figure
    
    Overlapping over the fence
    
    From the day I first met
    
    Taken away all of my heart
    
    You who wear a fleeting air somewhere
    
    I had lonely eyes
    
    Always with Tic Tac
    
    How many times in the ringing world
    
    To the noisy voice
    
    Even if tears are about to spill
    
    Mundane joy I'm sure you can find it
    
    To you who can't laugh in noisy days
    
    Have a dazzling tomorrow as long as you can think of
    
    Before falling on a night that doesn't dawn
    
    Grab my hand
    
    Even the days I locked up because I wanted to forget
    
    I'll melt it with the warmth I hug
    
    Not scary until the sun rises
    
    Let's be two
    
    I can only see you
    
    I hate you staring at something
    
    Like falling in love as if you were in love
    
    I hate that kind of face
    
    I want to believe but I can't believe it
    
    I'm sure what happened
    
    There are many from now on
    
    I'm angry and crying
    
    Still, I'm sure someday we'll be sure
    
    I believe you can understand each other
    
    I'm tired because I don't like it anymore
    
    You who shake my hand out to me
    
    I'm tired because I don't like it anymore
    
    I really want to say that too
    
    You see, with Tic Tac
    
    How many times in the ringing world
    
    I can't reach any of the words I prepared for you
    
    "I want to end it"
    
    When I was caught and put it into words
    
    You laughed for the first time
    
    I couldn't laugh in the noisy days
    
    You are beautiful in my eyes
    
    Even the tears that spilled on the night that didn't dawn
    
    Melting into your smile
    
    I was crying in the same days
    
    You gently invite to the end
    
    Like sinking, like melting
    
    The soaked fog clears
    
    In the days I was trapped because I wanted to forget
    
    Take your hand for reaching out
    
    The cool breeze blows through now like swimming in the sky
    
    Don't let go of your hands
    
    Two people are running out at night now
    """
))

translation

from transformers import pipeline
path = r"E:\HuggingFace\transformers\Helsinki-NLP\opus-mt-zh-en"
translator = pipeline("translation",model=path)
print(translator("我爱你"))

from transformers import pipeline
path = r"E:\HuggingFace\transformers\Helsinki-NLP\opus-mt-en-zh"
translator = pipeline("translation",model=path)
print(translator("I love you"))