pytorch 实现GPT2

papers

Gaussian Error Linear Units
translate to chinese

Attention Is All You Need
translate to chinese

Improving Language Understanding by Generative Pre-Training
translate to chinese

Language Models are Unsupervised Multitask Learners
translate to chinese

write story

input text

I love my motherland.

output

She saved my life."

The statement ends with the following line: "When I say I'm sorry, your daughter was killed."<|endoftext|>"I'd like to give you an insight into my family history," Trump said of the elder Johnson, saying that he was called into their home twice during a visit to Washington in the summer of 1996. "For some reason it was just in the background and that was it, but during that time I became very focused on the world and I decided, like a lot of people before me, that I was doing a lot of things that I never dreamed I would. These is a young woman that I started off being very conscious of but also knew where she belonged." "And then things became really serious over the summer," Trump continued. "And I started to pick things up and I've always felt there was a certain little thing about them that allowed me, in my life, to get really serious about that — or to feel really deep about those things and to really start, like, making connections if you will with what my family is trying to do, even for people and their kids and my brother, a lot of people were like, 'I don't know how, you do'. But just like there was a small corner of my life where I felt I had a certain way of going about everything."<|endoftext|>The United States government is using a law to force anyone to sign up for immigration legal aid who doesn't fit a comprehensive health insurance plan. The law has been passed in eight states last year to allow folks to skip insurance for themselves and their dependent children. The first step in establishing legal identification takes place a couple of weeks prior to next month's vote, and likely could pass, Sen. Patty Murray told reporters Tuesday at a Senate hearing on immigration reform.

Advertisement

If the law does pass, it would essentially extend the stay so you don't have to pay for essential health care, including those for your children. As many people point out, Medicaid in some other states is already in place to cover nonpregnant people, so not as many Americans may be getting adequate insurance coverage through that program as required by Federal law. The only thing a states' version of the law will do is tell people who might come to the country legally that the law's health care requirements apply.

I will probably be able to find a lawyer as quickly as possible to bring this issue to the attention of the House, and the Senate, and I will take the opportunity to do

Dataset

use the dataset ROCStories

run directly

show.py line:42 ***** Eval results *****
show.py line:44 eval_accuracy = 0.5531801175841796
show.py line:44 eval_loss = 0.6910396269244007
show.py line:44 train_loss = 0.0

run finetuned

show.py line:42 ***** Eval results *****
show.py line:44 eval_accuracy = 0.8273650454302512
show.py line:44 eval_loss = 0.36625397409129345
show.py line:44 train_loss = 3.138157744692941

这个结果不是很好，我怀疑我的实现有点问题，
gpt2 的eval accuracy 并没有gpt的结果好。
我并没有使用special tokens 在gpt2中
下面是gpt的结果

show.py line:42 ***** Eval results *****
show.py line:44 eval_accuracy = 0.863174772848744
show.py line:44 eval_loss = 0.31887995107815814
show.py line:44 train_loss = 3.087455103540013

随着训练轮数的增加，gpt2的eval accuracy 依然没有超过gpt。

github

https://github.com/darr/gpt2