因为入门了NLP,今天要用到bert,必须得有transformers`
# pip install transformers
from transformers import AutoTokenizer
#使用预训练模型 bert-base-uncased,模型内容详见https://huggingface.co/bert-base-uncased
# 分词器,词典
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
train_encoding = tokenizer(train_df['text'].to_list()[:], truncation=True, padding=True, max_length=512)
test_encoding = tokenizer(test_df['text'].to_list()[:], truncation=True, padding=True, max_length=512)
然后就没安装嘛,用pip install transformers报了错
Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determi
后来因为我电脑是CPU,没有GPU,所以说我换了一个命令
pip install transformers[torch]
后面又给我报错:
Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determi
还是这个错,我就改了命令:
pip install transformers --ignore-installed
还是报错:
Could not install packages due to an OSError: [WinError 5] 拒绝访问。: ‘D:\Anaconda3\Lib\site-packages\numpy\.libs\libopenblas.XWYDX2IKJW2NMTWSFYNGFUWKQU3LYTCZ.gfortran-win_amd64.dll’
Consider using the --user
option or check the permissions.
pip install --user transformers --ignore-installed
好像,好了
Successfully installed certifi-2022.6.15 charset-normalizer-2.1.0 colorama-0.4.5 filelock-3.7.1 huggingface-hub-0.8.1 idna-3.3 importlib-metadata-4.12.0 numpy-1.21.6 packaging-21.3 pyparsing-3.0.9 pyyaml-6.0 regex-2022.7.25 requests-2.28.1 tokenizers-0.12.1 tqdm-4.64.0 transformers-4.21.0 typing-extensions-4.3.0 urllib3-1.26.11 zipp-3.8.1
。。。。
慢慢来吧,NLP现在才只是一个开始。。。