【pkuseg】由于网络策略组织下载请求，因此直接在github中下载细分领域模型medicine

最新推荐文章于 2024-05-31 15:01:43 发布

是Yu欸

最新推荐文章于 2024-05-31 15:01:43 发布

阅读量1.3k

点赞数 29

分类专栏： # 文本处理与摘要文章标签： github pkuseg 分词 nlp 网络机器学习

本文链接：https://blog.csdn.net/wtyuong/article/details/138712919

版权

文本处理与摘要专栏收录该内容

24 篇文章 5 订阅

订阅专栏

【pkuseg】由于网络策略组织下载请求，因此直接在github中下载细分领域模型medicine

写在最前面

请添加图片描述

🌈你好呀！我是是Yu欸 🌌 2024每日百字篆刻时光，感谢你的陪伴与支持 ~ 🚀 欢迎一起踏上探险之旅，挖掘无限可能，共同成长！

写在最前面

运行代码时，报错。

代码示例2：细领域分词（如果用户明确分词领域，推荐使用细领域模型分词）

import pkuseg

seg = pkuseg.pkuseg(model_name='medicine')  # 程序会自动下载所对应的细领域模型
text = seg.cut('我爱北京天安门')              # 进行分词
print(text)

解决方案

Downloading: “https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip” to /home/tjh/.pkuseg/medicine.zip

搜索下载https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip
然后解压，并将压缩包+解压后文件夹，放到文件夹/home/tjh/.pkuseg/medicine.zip下。
然后就完美解决啦！！！嘿嘿

记得解压，否则报错：

Traceback (most recent call last):
  File "/home/tjh/wyt/CT-base/4-2.Seq2seq_Att/load_data.py", line 41, in <module>
    seg = pkuseg.pkuseg(model_name='medicine')
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/pkuseg/__init__.py", line 239, in __init__
    self.postprocesser = Postprocesser(None, other_names)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/pkuseg/__init__.py", line 148, in __init__
    with open(other_name, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/tjh/.pkuseg/medicine/medicine_dict.pkl'

在这里插入图片描述

pkuseg是什么？

PKUSEG 是一个由北京大学开发的多领域中文分词工具。这个工具在分词精度上比很多流行的中文分词工具都要优秀，特别是在处理不同领域的文本时，因为它支持对特定领域的文本进行训练和优化。下面是一些PKUSEG的关键特性：

多领域分词：PKUSEG 支持多种领域的分词，包括但不限于通用文本、网络文本、旅游、新闻等，可以根据特定的领域数据进行优化。
高精度：该工具使用了基于深度学习的模型，与传统的基于规则和统计的分词工具相比，能更好地处理复杂的中文文本和歧义现象。
易于使用：PKUSEG 提供了简洁的 API，便于在Python环境中直接使用。用户可以很容易地集成和部署在自己的应用程序中。
自定义模型训练：用户可以使用自己的标注数据来训练专门的分词模型，进一步提升分词效果在特定应用场景中的表现。
开源：PKUSEG 是开源软件，任何人都可以使用、修改和分发，这为开发者提供了极大的灵活性和扩展性。

使用PKUSEG进行中文分词非常简单，一般包括安装、模型选择、文本分词等步骤。这样的工具非常适合需要处理具体领域中文文本的开发者和研究人员。

报错原因

你的代码试图从GitHub下载一个名为"medicine.zip"的文件，但是在SSL证书验证过程中出现了错误。这可能是由于你的网络环境中的SSL证书问题，或者是因为你的Python环境中缺少某些必要的证书。

这里我的问题是第一个，因此直接下载后复制到该区域即可。

你可以尝试以下几种解决方案：

如果你在公司或学校网络环境中，可能存在网络策略阻止了你的下载请求。你可以尝试在其他网络环境中运行你的代码。
你可以尝试更新你的Python环境中的certifi库。这个库包含了Python需要的SSL证书。你可以在终端中运行以下命令来更新这个库：

pip install --upgrade certifi

如果以上方法都不能解决问题，你可以尝试禁用SSL证书验证。但是请注意，这种方法可能会带来安全风险。你可以在你的代码中找到所有的requests.get或requests.post调用，然后添加一个参数verify=False来禁用SSL证书验证。例如：

requests.get(url, verify=False)

请注意，这种方法可能会导致你的代码在某些环境中无法正常工作，因为它依赖于你的网络环境是否允许未经验证的SSL连接。

报错详情

(pytorch1.12_gpu) tjh@tjh-SYS:~/wyt/CT-base$ /home/tjh/miniconda3/envs/pytorch1.12_gpu/bin/python /home/tjh/wyt/CT-base/4-2.Seq2seq_Att/load_data.py
cuda
Downloading: “https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip” to /home/tjh/.pkuseg/medicine.zip
Traceback (most recent call last):

  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'objects.githubusercontent.com'. (_ssl.c:1122)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'objects.githubusercontent.com'. (_ssl.c:1122)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objects.githubusercontent.com', port=443): Max retries exceeded with url: /github-production-release-asset-2e65be/143589809/c6d88a00-360f-11e9-9b5e-57de5648f695?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240511%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240511T053934Z&X-Amz-Expires=300&X-Amz-Signature=8d5146d02a102c051952c64a2a73d2b6e58486d736468807511cb2e4c7c82ad0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=143589809&response-content-disposition=attachment%3B%20filename%3Dmedicine.zip&response-content-type=application%2Foctet-stream (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'objects.githubusercontent.com'. (_ssl.c:1122)")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tjh/wyt/CT-base/4-2.Seq2seq_Att/load_data.py", line 41, in <module>
    seg = pkuseg.pkuseg(model_name='medicine')
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/pkuseg/__init__.py", line 207, in __init__
    download_model(config.model_urls[model_name], config.pkuseg_home, config.model_hash[model_name])
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/pkuseg/download.py", line 38, in download_model
    _download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/pkuseg/download.py", line 44, in _download_url_to_file
    u = urlopen(url, stream=True, timeout=5)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/sessions.py", line 725, in send
    history = [resp for resp in gen]
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/sessions.py", line 725, in <listcomp>
    history = [resp for resp in gen]
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/tjh/miniconda3/envs/pytorch1.12_gpu/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='objects.githubusercontent.com', port=443): Max retries exceeded with url: /github-production-release-asset-2e65be/143589809/c6d88a00-360f-11e9-9b5e-57de5648f695?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240511%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240511T053934Z&X-Amz-Expires=300&X-Amz-Signature=8d5146d02a102c051952c64a2a73d2b6e58486d736468807511cb2e4c7c82ad0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=143589809&response-content-disposition=attachment%3B%20filename%3Dmedicine.zip&response-content-type=application%2Foctet-stream (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'objects.githubusercontent.com'. (_ssl.c:1122)")))