我一般学习新的框架或工具库喜欢看着官方文档来码代码,所以本文就以这个方式介绍下自己常用的一些机器学习工具库。
机器学习
Numpy
NumPy是一个用于数值计算的库,提供了多维数组和矩阵操作,适用于处理大规模数学计算和数据处理。
numpyhttps://www.numpy.org.cn/reference/
import numpy as np
a = np.arange(12)
print(a.view())
print('\nafter reshape...\n')
a.shape = 3, 4
print(a.view())
boardcast = a**3
print('\n广播')
print(boardcast)
Output:
[ 0 1 2 3 4 5 6 7 8 9 10 11]
after reshape...
[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
广播
[[ 0 1 8 27]
[ 64 125 216 343]
[ 512 729 1000 1331]]
Pandas
Pandas是一个数据分析库,提供了数据结构和数据分析工具,适用于处理格式化的数据文件。
pandashttps://www.pypandas.cn/docs/
import pandas as pd
s = pd.Series([1, 3, 5, np.nan, 6, 8])
print('Series\n')
print(s)
df = pd.DataFrame({
'A': 1.,
'B': pd.Timestamp('20130102'),
'C': pd.Series(1, index=list(range(4)), dtype='float32'),
'D': np.array([3] * 4, dtype='int32'),
'E': pd.Categorical(["test", "train", "test", "train"]),
'F': 'foo'})
print('\nDataFrame\n')
print(df)
Output:
Series:
0 1.0
1 3.0
2 5.0
3 NaN
4 6.0
5 8.0
dtype: float64
DataFrame:
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
2 1.0 2013-01-02 1.0 3 test foo
3 1.0 2013-01-02 1.0 3 train foo
Gensim
Gensim是一个用于主题建模和文档相似性的库,适用于处理文本数据。
gensimhttps://radimrehurek.com/gensim/apiref.html#api-reference
text_corpus = [
"Human machine interface for lab abc computer applications",
"A survey of user opinion of computer system response time",
"The EPS user interface management system",
"System and human system engineering testing of EPS",
"Relation of user perceived response time to error measurement",
"The generation of random binary unordered trees",
"The intersection graph of paths in trees",
"Graph minors IV Widths of trees and well quasi ordering",
"Graph minors A survey",
]
# Create a set of frequent words
stoplist = set('for a of the and to in'.split(' '))
# Lowercase each document, split it by white space and filter out stopwords
texts = [[word for word in document.lower().split() if word not in stoplist]
for document in text_corpus]
# Count word frequencies
from collections import defaultdict
frequency = defaultdict(int)
for text in texts:
for token in text:
frequency[token] += 1
# Only keep words that appear more than once
processed_corpus = [[token for token in text if frequency[token] > 1] for text in texts]
print(processed_corpus)
from gensim import corpora
dictionary = corpora.Dictionary(processed_corpus)
print(dictionary)
print(dictionary.token2id)
new_doc = "Human response interaction"
new_vec = dictionary.doc2bow(new_doc.lower().split())
print(new_vec)
Output:
processed_corpus:
[['human', 'interface', 'computer'], ['survey', 'user', 'computer', 'system', 'response', 'time'], ['eps', 'user', 'interface', 'system'], ['system', 'human', 'system', 'eps'], ['user', 'response', 'time'], ['trees'], ['graph', 'trees'], ['graph', 'minors', 'trees'], ['graph', 'minors', 'survey']]
dictionary:
Dictionary<12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...>
dictionary.token2id:
{'computer': 0, 'human': 1, 'interface': 2, 'response': 3, 'survey': 4, 'system': 5, 'time': 6, 'user': 7, 'eps': 8, 'trees': 9, 'graph': 10, 'minors': 11}
new_vec:
[(1, 1), (3, 1)]
深度学习 - torch
Torch是一个深度学习框架,提供了构建和训练神经网络所需的工具和API,适用于处理大规模数据和复杂模型。
torchhttps://pytorch-cn.readthedocs.io/zh/latest/
大语言模型 - transformer
Transformer是Huggingface公司维护的大语言模型,适用于处理自然语言任务,如文本分类、翻译和生成等。这些工具库都具有易于使用、高效、灵活和可扩展的特点,可以帮助开发者快速构建和训练机器学习模型,解决实际问题。
transformer(预训练大模型,需要翻墙)https://huggingface.co/transformers
transformer源码https://github.com/huggingface/transformers/blob/main/README_zh-hans.md