huggingface - pipeline - translate 记录

小田_

于 2023-11-30 19:58:03 发布

阅读量417

点赞数 9

分类专栏： NLP 文章标签： huggingface

本文链接：https://blog.csdn.net/weixin_45390999/article/details/134713683

版权

NLP 专栏收录该内容

26 篇文章 5 订阅

订阅专栏

文章目录

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

import os ,sys  
from transformers import pipeline

model_checkpoint = "Helsinki-NLP/opus-mt-zh-en"
translator = pipeline("translation", model=model_checkpoint)

def translate_arr(arr): 
    ret = translator(arr) 
    # print(ret) 

    arr = []
    for d in ret:
        arr.append(d['translation_text'])

    # print(arr) 
    return arr

模型存储位置：
~/.cache/huggingface/hub/models--Helsinki-NLP--opus-mt-th-en

查看内容

models--Helsinki-NLP--opus-mt-th-en $ tree
.
├── blobs
│   ├── 04d12b26d23b7fdafb05e6e6b55c0d77eeb5c0553b009d7e8cd5c5027103f05a
│   ├── 1f32eb49d2060249ed6cd1f1cad366e341852b23
│   ├── 4f1f84ae80ee81e09d87f2a2deecf284e2df1648
│   ├── 8ad305d29e9707566ec7bff50d042e2704408660
│   ├── b6177e8516f6f5242a8a6658814a4945366778ab
│   ├── d584343f48cfc150e5f5c6855b683227689f399d
│   └── e755cf32f77dcdf0f19ccb8a8409ac8576459d0e
├── refs
│   └── main
└── snapshots
    ├── 0fff80be1c3f40d6180efd59f566ece956ace689
    │   ├── config.json -> ../../blobs/e755cf32f77dcdf0f19ccb8a8409ac8576459d0e
    │   ├── generation_config.json -> ../../blobs/d584343f48cfc150e5f5c6855b683227689f399d
    │   ├── pytorch_model.bin -> ../../blobs/04d12b26d23b7fdafb05e6e6b55c0d77eeb5c0553b009d7e8cd5c5027103f05a
    │   ├── source.spm -> ../../blobs/4f1f84ae80ee81e09d87f2a2deecf284e2df1648
    │   ├── target.spm -> ../../blobs/1f32eb49d2060249ed6cd1f1cad366e341852b23
    │   ├── tokenizer_config.json -> ../../blobs/8ad305d29e9707566ec7bff50d042e2704408660
    │   └── vocab.json -> ../../blobs/b6177e8516f6f5242a8a6658814a4945366778ab
    └── f8ac5635d92074f85ed45730098eca069011d1dd
        ├── pytorch_model.bin -> ../../blobs/04d12b26d23b7fdafb05e6e6b55c0d77eeb5c0553b009d7e8cd5c5027103f05a
        ├── source.spm -> ../../blobs/4f1f84ae80ee81e09d87f2a2deecf284e2df1648
        ├── target.spm -> ../../blobs/1f32eb49d2060249ed6cd1f1cad366e341852b23
        ├── tokenizer_config.json -> ../../blobs/8ad305d29e9707566ec7bff50d042e2704408660
        └── vocab.json -> ../../blobs/b6177e8516f6f5242a8a6658814a4945366778ab

6 directories, 20 files