这是第一段代码，关于bertopic主题建模“# 加载原始文本数据（仍需用于主题表示）
df = pd.read_csv(‘tokenized_abstract.csv’, encoding=‘utf-8’)
sentences = df[‘Abstract’].tolist()
print('文本条数: ', len(sentences))
print('预览第一条: ', sentences[0])
检查缺失值
print(“缺失值数量:”, df[‘Abstract’].isna().sum())
检查非字符串类型
non_str_mask = df[‘Abstract’].apply(lambda x: not isinstance(x, str))
print(“非字符串样本:\n”, df[non_str_mask][‘Abstract’].head())
vectorizer_model = Nonefrom sentence_transformers import SentenceTransformer
Step 1 - Extract embeddings
embedding_model = SentenceTransformer(“C:\Users\18267\.cache\huggingface\hub\models–sentence-transformers–all-mpnet-base-v2\snapshots\9a3225965996d404b775526de6dbfe85d3368642”)
embeddings = np.load(‘clean_emb_last.npy’)
print(f"嵌入的形状: {embeddings.shape}")
Step 2 - Reduce dimensionality
umap_model = UMAP(n_neighbors=7, n_components=10, min_dist=0.0, metric=‘cosine’,random_state=42)
Step 3 - Cluster reduced embeddings
hdbscan_model = HDBSCAN(min_samples=7, min_cluster_size=60,metric=‘euclidean’,
cluster_selection_method=‘eom’,
prediction_data=True)
Step 4 - Tokenize topics
Combine custom stop words with scikit-learn’s English stop words
custom_stop_words = [‘h2’, ‘storing’, ‘storage’, ‘include’, ‘comprise’,
‘utility’, ‘model’, ‘disclosed’, ‘embodiment’, ‘invention’, ‘prior’, ‘art’,
‘according’, ‘present’, ‘method’, ‘system’, ‘device’, ‘may’, ‘also’, ‘use’,
‘used’, ‘provide’, ‘wherein’, ‘configured’, ‘predetermined’, ‘plurality’,
‘comprising’, ‘consists’, ‘following’, ‘characterized’, ‘claim’, ‘claims’,
‘said’, ‘first’, ‘second’, ‘third’, ‘fourth’, ‘fifth’, ‘one’, ‘two’, ‘three’,‘hydrogen’]
Create combined stop words set
all_stop_words = set(custom_stop_words).union(ENGLISH_STOP_WORDS)
vectorizer_model = CountVectorizer(stop_words=list(all_stop_words))
Step 5 - Create topic representation
ctfidf_model = ClassTfidfTransformer()
All steps together
topic_model = BERTopic(
embedding_model=embedding_model, # Step 1 - Extract embeddings
umap_model=umap_model, # Step 2 - Reduce dimensionality
hdbscan_model=hdbscan_model, # Step 3 - Cluster reduced embeddings
vectorizer_model=vectorizer_model, # Step 4 - Tokenize topics
ctfidf_model=ctfidf_model, # Step 5 - Extract topic words
top_n_words=50
)# 拟合模型
topics, probs = topic_model.fit_transform(documents=sentences, # 仍需提供文档用于主题词生成
embeddings=embeddings # 注入预计算嵌入)
)
获取主题聚类信息
topic_info = topic_model.get_topic_info()
print(topic_info)”
最新发布
本文探讨了古老的邮件编码问题，重点聚焦于UUencode。UUencode是一种早期在Unix邮件系统中使用的二进制到文字编码方式，用于将二进制数据转化为可打印字符。它将每个3字节的数据分成4个6位组，并转换为32-95之间的ASCII字符，以确保可打印性。虽然与Base64相似，但UUencode有其独特的转码规则。
最低0.47元/天 解锁文章
扫一扫
3757
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
