热爱Coding
自然语言处理三大顶会审稿人
展开
-
pytorch 检查GPU可用
【代码】pytorch 检查GPU可用。原创 2023-11-28 11:07:43 · 105 阅读 · 0 评论 -
huggingface/transformers 用Trainer 和 不用Trainer
【代码】huggingface/transformers 用Trainer 和 不用Trainer。原创 2023-10-19 15:53:07 · 113 阅读 · 0 评论 -
huggingface/transformers 确保模型利用GPU
【代码】huggingface/transformers 确保模型利用GPU。原创 2023-09-05 16:49:58 · 711 阅读 · 0 评论 -
huggingface/transformers 预训练自己的数据
huggingface/transformers 会自动加载 能对应上的 参数。原创 2023-08-31 17:12:06 · 151 阅读 · 0 评论 -
AutoModelForSequenceClassification 得到embedding
【代码】AutoModelForSequenceClassification 得到embedding。原创 2023-08-25 20:45:16 · 106 阅读 · 0 评论 -
不预训练的话,用huggingface/transformers比google-research/bert的效果好2-4%
用于文本分类,正确率高2-4%原创 2023-08-01 11:00:44 · 85 阅读 · 0 评论 -
报错:Module ‘XXX‘ doesn‘t exist on the Hugging Face Hub
【代码】报错:Module ‘accuracy‘ doesn‘t exist on the Hugging Face Hub。原创 2023-08-01 10:04:40 · 2008 阅读 · 5 评论 -
huggingface/transformers tokenizer报错 list index out of range
【代码】huggingface/transformers tokenizer报错 list index out of range。原创 2023-06-28 10:18:07 · 314 阅读 · 0 评论 -
huggingface/transformers TFAutoModelForSeq2SeqLM 输出 embedding
【代码】huggingface/transformers TFAutoModelForSeq2SeqLM 输出 embedding。原创 2023-06-27 14:51:44 · 272 阅读 · 0 评论 -
LOMO: LOw-Memory Optimization 阅读笔记
文章贡献:减少对GPU内存的占用,从而能让GPU容纳更大的模型参数。原创 2023-06-26 10:46:08 · 94 阅读 · 0 评论 -
ImportError: cannot import name ‘skip_init‘ from ‘torch.nn.utils‘
torch==1.10.0 版本可以。问题所在:pytorch版本不对,原创 2023-06-19 12:00:04 · 1634 阅读 · 0 评论 -
symbol cudaGraphInstantiateWithFlags, version libcudart.so.11.0 not defined
问题所在:pytorch和cuda版本不匹配。原创 2023-06-19 11:47:27 · 1148 阅读 · 1 评论 -
deepspeed ZeRO-Inference 可在1-GPU上推理~100B的大模型
https://www.deepspeed.ai/2022/09/09/zero-inference.html原理:原创 2023-06-18 10:22:23 · 224 阅读 · 0 评论 -
huggingface/transformers,tokenizer出的attention_mask的坑
则tokenizer返回的。则tokenizer返回的。原创 2023-06-15 11:53:35 · 353 阅读 · 0 评论 -
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
huggingface/transformers 的 tokenizer 报的错,报错原因,数据里有None值。原创 2023-06-15 09:06:21 · 377 阅读 · 0 评论 -
把huggingface/transformers的模型文件下载到本地
python代码 自动下载模型实现和参数。如果你的网络环境较差,下载模型参数可能会花费较长时间甚至失败。此时可以先将模型下载到本地,然后从本地加载。参考 https://github.com/THUDM/ChatGLM-6B 的README。以chatglm-6b为例,然后运行。先安装Git LFS。原创 2023-06-14 15:52:09 · 2164 阅读 · 0 评论 -
contrastive loss 的高级代码实现 (pytorch)
LARGE_NUM = 1e9 def _contrastive_loss_forward(self, hidden1: torch.Tensor, hidden2: torch.Tensor, hidden_norm: bool = True, ...原创 2021-08-17 10:55:46 · 4303 阅读 · 0 评论 -
把 kmeans聚类 融入深度模型 --> KmeansAttention
https://github.com/lucidrains/routing-transformer/blob/master/routing_transformer/routing_transformer.py原创 2021-08-09 10:43:49 · 413 阅读 · 0 评论 -
pytorch add_
# a = a + 4 * 5import torcha = torch.tensor([1,2,3])a.data.add_(torch.tensor(4),torch.tensor(5))print(a) # tensor([21, 22, 23])原创 2020-09-22 20:40:05 · 1240 阅读 · 0 评论 -
pytorch addcdiv 和 addcdiv_
# a = a + 4 / 2import torcha = torch.tensor([1,2,3])a.addcdiv(torch.tensor(4),torch.tensor(2))print(a) # tensor([1, 2, 3]) # 值不加a.data.addcdiv_(torch.tensor(4),torch.tensor(2))print(a) # tensor([3, 4, 5]) # 值加a = a.addcdiv(torch.tensor(4),torch.t原创 2020-09-22 20:37:27 · 983 阅读 · 0 评论 -
sh run安装cuda失败,Finished with code: 256
要先禁用nouveau保证lsmod |grep nouveau不显示信息原创 2020-06-20 21:01:27 · 17311 阅读 · 3 评论 -
pytorch pad 实例
import torchtensor = torch.Tensor([[[1,1],[2,2],[3,3]],[[4,4],[5,5],[6,6]]])print(tensor.shape)print(tensor)pad_tensor = torch.constant_pad_nd(tensor,(0,0,0,2))print(pad_tensor.shape)print(pad_tensor)print结果:torch.Size([2, 3, 2])tensor([[[1., 1.]原创 2020-05-09 19:30:15 · 902 阅读 · 0 评论 -
multi-label分类,loss一直增大
label为[batch_size, num_class]logits为[batch_size, num_class]每个label为比如[0,0,1,0,0,0,1,0,1,0],就是10类有3类正确不能用tf.nn.softmax_cross_entropy_with_logitsPytorch使用torch.nn.BCElossTensorflow使用tf.losses.sigmo...原创 2020-02-25 20:13:35 · 755 阅读 · 0 评论 -
一个pytorch的pointer net实现
https://github.com/pcyin/tranX/blob/master/model/pointer_net.py原创 2019-12-16 11:20:29 · 641 阅读 · 0 评论 -
一个带copy机制的seq2seq的pytorch实现
https://github.com/pcyin/tranX/blob/master/model/seq2seq_copy.py原创 2019-12-16 11:19:38 · 855 阅读 · 0 评论 -
一种输入[batch, seq_len1, hidden_dim]输出[batch, seq_len2, hidden_dim]的self-attention的pytorch实现
class Attention(nn.Module): """ inputs是[batch, seq_len1, hidden_dim] labels_num是seq_len2 """ def __init__(self, labels_num, hidden_size): super(Attention, self).__init__()...原创 2019-12-16 11:17:23 · 712 阅读 · 0 评论 -
pytorch 三维one-hot tensor的制作
import torchbatch_size = 2sequence_len = 3hidden_dim = 5x = torch.zeros(batch_size, sequence_len, hidden_dim).scatter_(dim=-1, index=torch.LongTensor([[[2],[2],[1]]...原创 2019-10-14 09:24:30 · 1482 阅读 · 0 评论 -
pytorch 报错 RuntimeError: Invalid index in scatter at
很大可能是因为index的值超出了范围,比如import torchbatch_size = 2hidden_dim = 5x = torch.zeros(batch_size, hidden_dim).scatter_(dim=-1, index=torch.LongTensor([[2],[1]]), ...原创 2019-10-14 09:20:39 · 2614 阅读 · 0 评论 -
pytorch 以cosine为loss训练
其中rep_a为[batch_size,hidden_dim]rep_b为[batch_size,hidden_dim]labels为[batch_size]摘自https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/models/TransformerModel.py...原创 2019-08-05 10:47:50 · 5083 阅读 · 0 评论 -
OpenNMT做端对端的接口使用
安装:其中requirements.txt里安装的是很新的torchtextgit clone --branch 0.9.1 https://github.com/OpenNMT/OpenNMT-py.gitcd OpenNMT-pypip install -r requirements.txt cd ..预处理:其中src-train.txt和tgt-train.txt为原始英文...原创 2019-07-09 15:16:29 · 1295 阅读 · 8 评论 -
pytorch multinomial 报错 device-side assert triggered
/pytorch/aten/src/THC/THCTensorRandom.cuh:187: void sampleMultinomialOnce(long *, long, int, T *, T *, int, int) [with T = float, AccT = float]: block: [6,0,0], thread: [5,0,0] Assertion `THCNumerics&...原创 2019-06-14 16:45:43 · 1521 阅读 · 0 评论 -
pytorch的reinforce算法 官方文档
http://pytorch.org/docs/0.3.0/distributions.htmlprobs = policy_network(state)m = Categorical(probs)action = m.sample() # 抽样一个actionnext_state, reward = env.step(action) # 得到一个rewardloss = -m.l原创 2018-01-05 11:16:05 · 2967 阅读 · 0 评论 -
Multiple Negatives Ranking Loss 的pytorch实现
摘自https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/losses.py原创 2019-08-05 11:24:48 · 2066 阅读 · 0 评论 -
pytorch 欧式距离 euclidean distance 实现
import torch.nn.functional as Fdistance = F.pairwise_distance(rep_a, rep_b, p=2)其中rep_a和rep_a为[batch_size,hidden_dim]原创 2019-08-21 15:11:26 · 19955 阅读 · 0 评论 -
pytorch attend操作 代码
# seq: [batch,sel_len,hidden_dim]# cond: [batch,hidden_dim]# lens: [batch] def attend(seq, cond, lens): """ attend over the sequences `seq` using the condition `cond`. """ scores = ...原创 2019-09-09 08:36:35 · 340 阅读 · 0 评论 -
sqlova 代码走读
输入BERT的方式 https://github.com/naver/sqlova/blob/master/sqlova/utils/utils_wikisql.py在generate_inputs方法[CLS] question_word_1,question_word_2,...question_word_n [SEP] header_1 [SEP] header_2 [SEP] ... ...原创 2019-09-10 11:06:55 · 614 阅读 · 0 评论 -
pytorch one-hot tensor的制作
import torchbatch_size = 2hidden_dim = 5x = torch.zeros(batch_size, hidden_dim).scatter_(dim=-1, index=torch.LongTensor([[2],[1]]), va...原创 2019-10-11 16:30:59 · 583 阅读 · 0 评论 -
pytorch的Entropy Minimization (EM) 的实现
# p_logit: [batch,class_num]def entropy_loss(p_logit): p = F.softmax(p_logit, dim=-1) return -1 * torch.sum(p * F.log_softmax(p_logit, dim=-1)) / p_logit.size()[0]原创 2019-05-16 14:49:08 · 2127 阅读 · 5 评论 -
pytorch 的KL divergence的实现
import torch.nn.functional as F# p_logit: [batch,dim0]# q_logit: [batch,dim0]def kl_categorical(p_logit, q_logit): p = F.softmax(p_logit, dim=-1) _kl = torch.sum(p * (F.log_softmax(p_logit...原创 2019-05-16 14:23:56 · 9243 阅读 · 0 评论 -
pytorch 固定部分参数训练
需要自己过滤optimizer.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)另外,如果是Variable,则可以初始化时指定j = Variable(torch.randn(5,5), requires_grad=True)但是如果是m = nn.Linear(10,10)...原创 2018-03-29 11:16:17 · 34047 阅读 · 19 评论