tensorflow - tensor2tensor - v1.0.12

于 2020-05-30 08:23:16 发布
阅读量2.3k 收藏 2
点赞数 2
分类专栏： 20240202 文章标签： tensorflow tensor2tensor v1.0.12
世上没有白读的书，每一页都算数。
本文链接：https://blog.csdn.net/chengyq116/article/details/106436539
版权

tensorflow - tensor2tensor - v1.0.12

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
深度学习模型和数据集的库，旨在使深度学习更易于访问并加速 ML 研究。tensor2tensor 是一个库，里面封装了很多的模型。

https://github.com/tensorflow/tensor2tensor
https://github.com/tensorflow/tensor2tensor/releases
https://github.com/tensorflow/tensor2tensor/tags
https://github.com/tensorflow/tensor2tensor/releases/tag/v1.0.12
https://github.com/tensorflow/tensor2tensor/tree/v1.0.12
https://github.com/tensorflow/tensorflow/tree/r1.4
https://tensorflow.google.cn/install/source

在这里插入图片描述

T2T is a modular and extensible library and binaries for supervised learning with TensorFlow and with support for sequence tasks. It is actively used and maintained by researchers and engineers within the Google Brain team. You can read more about Tensor2Tensor in the recent Google Research Blog post introducing it.
https://github.com/tensorflow/tensor2tensor/issues
https://github.com/tensorflow/tensor2tensor/blob/v1.0.12/CONTRIBUTING.md
https://gitter.im/tensor2tensor/Lobby

tensor2tensor 主要有 4 个步骤：生成数据、训练、解码、评分。

0. References

The Illustrated Transformer
https://jalammar.github.io/illustrated-transformer/
Kyubyong - transformer
https://github.com/Kyubyong/transformer

1. T2T: Tensor2Tensor Transformers - v1.0.12

yongqiang@yongqiang:~$ ls
yongqiang@yongqiang:~$
yongqiang@yongqiang:~$ mkdir software
yongqiang@yongqiang:~$ mkdir tensorflow_work
yongqiang@yongqiang:~$ mkdir yongqiang
yongqiang@yongqiang:~$ ls
software  tensorflow_work  yongqiang
yongqiang@yongqiang:~$
yongqiang@yongqiang:~$ ll
total 8
drwxr-xr-x 1 yongqiang yongqiang  512 May 30 07:27 ./
drwxr-xr-x 1 root      root       512 Apr 30 17:52 ../
-rw------- 1 yongqiang yongqiang  396 May 30 07:26 .bash_history
-rw-r--r-- 1 yongqiang yongqiang  220 Apr 30 17:52 .bash_logout
-rw-r--r-- 1 yongqiang yongqiang 3771 Apr 30 17:52 .bashrc
drwxrwxrwx 1 yongqiang yongqiang  512 Apr 30 17:52 .cache/
drwx------ 1 yongqiang yongqiang  512 Apr 30 17:52 .config/
-rw-r--r-- 1 yongqiang yongqiang  807 Apr 30 17:52 .profile
-rw------- 1 yongqiang yongqiang    7 May  2 09:17 .python_history
-rw-r--r-- 1 yongqiang yongqiang    0 May  2 08:15 .sudo_as_admin_successful
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 software/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 tensorflow_work/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 yongqiang/
yongqiang@yongqiang:~$
yongqiang@yongqiang:~$ mkdir tensor2tensor
yongqiang@yongqiang:~$ cd tensor2tensor/
yongqiang@yongqiang:~/tensor2tensor$ mkdir v1_0_12
yongqiang@yongqiang:~/tensor2tensor$ ls
v1_0_12
yongqiang@yongqiang:~/tensor2tensor$ cd v1_0_12/
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$ ls
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$ git clone https://github.com/tensorflow/tensor2tensor.git
Cloning into 'tensor2tensor'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 32312 (delta 0), reused 5 (delta 0), pack-reused 32307
Receiving objects: 100% (32312/32312), 15.47 MiB | 46.00 KiB/s, done.
Resolving deltas: 100% (26143/26143), done.
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$ ls
tensor2tensor
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12$ cd tensor2tensor/
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ ll
total 52
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:49 ./
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:43 ../
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:49 .git/
-rw-rw-rw- 1 yongqiang yongqiang   310 May 30 07:49 .gitignore
-rw-rw-rw- 1 yongqiang yongqiang   779 May 30 07:49 .travis.yml
-rw-rw-rw- 1 yongqiang yongqiang   311 May 30 07:49 AUTHORS
-rw-rw-rw- 1 yongqiang yongqiang  1280 May 30 07:49 CONTRIBUTING.md
-rw-rw-rw- 1 yongqiang yongqiang   266 May 30 07:49 ISSUE_TEMPLATE.md
-rw-rw-rw- 1 yongqiang yongqiang 11358 May 30 07:49 LICENSE
-rw-rw-rw- 1 yongqiang yongqiang 19726 May 30 07:49 README.md
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:49 docs/
-rw-rw-rw- 1 yongqiang yongqiang    34 May 30 07:49 floyd.yml
-rw-rw-rw- 1 yongqiang yongqiang    14 May 30 07:49 floyd_requirements.txt
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:49 oss_scripts/
-rw-rw-rw- 1 yongqiang yongqiang  7866 May 30 07:49 pylintrc
-rw-rw-rw- 1 yongqiang yongqiang  3413 May 30 07:49 setup.py
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:49 tensor2tensor/
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

git branch - 查看本地所有分支 (结果列表中前面标 * 号表示本地当前使用的分支)。
git branch 命令，如果不加任何参数运行它，会得到当前所有分支的一个列表。注意 master 分支前的 * 字符：它代表现在检出的那一个分支 (也就是说，当前 HEAD 指针所指向的分支)。这意味着如果在这时候提交，master 分支将会随着新的工作向前移动。如果需要查看每一个分支的最后一次提交，可以运行 git branch -v 命令。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git branch
* master
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

git branch -a - 查看所有分支 (本地 + 远程)。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/revert-1726-master
  remotes/origin/revert-1749-revert-1726-master
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

git branch -r - 查看远程所有分支。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git branch -r
  origin/HEAD -> origin/master
  origin/master
  origin/revert-1726-master
  origin/revert-1749-revert-1726-master
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

Git 可以给仓库历史中的某一个提交打上标签，以示重要。比较有代表性的是人们会使用这个功能来标记发布结点 (v1.0、v2.0 等等)。

在 Git 中列出已有的标签非常简单，只需要输入 git tag (可带上可选的 -l 选项 --list)。这个命令以字母顺序列出标签，但是它们显示的顺序并不重要。

按照通配符列出标签需要 -l 或 --list 选项。如果你只想要完整的标签列表，那么运行 git tag 就会默认假定你想要一个列表，它会直接给你列出来， 此时的 -l或 --list 是可选的。如果你提供了一个匹配标签名的通配模式，那么 -l 或 --list 就是强制使用的。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git tag
1.3.2
v.1.12.0
v1.0.10
v1.0.11
v1.0.12
v1.0.13
v1.0.14
......
v1.9.0
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git tag -l
1.3.2
v.1.12.0
v1.0.10
v1.0.11
v1.0.12
v1.0.13
v1.0.14
......
v1.9.0
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git tag -l 'v1.0.*'
v1.0.10
v1.0.11
v1.0.12
v1.0.13
v1.0.14
v1.0.3
v1.0.4
v1.0.5
v1.0.6
v1.0.7
v1.0.8
v1.0.9
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

git show v1.0.12 - 查看 tag v1.0.12 的版本信息。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git show v1.0.12
commit 83ac9fb735930c4e1b3b14504e63c1c55b95da75 (tag: v1.0.12)
Merge: fbb6f9ae 54622a51
Author: Lukasz Kaiser <lukaszkaiser@users.noreply.github.com>
Date:   Fri Jul 7 17:30:39 2017 -0700

    Merge pull request #118 from lukaszkaiser/push

    1.0.12

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

git checkout tag_name - git 切换到某个 tag，git 可能会提示你当前处于一个 detached HEAD 状态。因为 tag 相当于是一个快照，是不能更改它的代码的。如果要在 tag 代码的基础上做修改，你需要新建一个分支。

yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git checkout v1.0.12
Note: checking out 'v1.0.12'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 83ac9fb7 Merge pull request #118 from lukaszkaiser/push
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ git status
HEAD detached at v1.0.12
nothing to commit, working tree clean
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ ls
AUTHORS  CONTRIBUTING.md  LICENSE  README.md  setup.py  tensor2tensor
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$ ll
total 32
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 08:10 ./
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 07:43 ../
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 08:10 .git/
-rw-rw-rw- 1 yongqiang yongqiang   251 May 30 08:10 .gitignore
-rw-rw-rw- 1 yongqiang yongqiang   293 May 30 08:10 AUTHORS
-rw-rw-rw- 1 yongqiang yongqiang   969 May 30 08:10 CONTRIBUTING.md
-rw-rw-rw- 1 yongqiang yongqiang 11358 May 30 07:49 LICENSE
-rw-rw-rw- 1 yongqiang yongqiang  9069 May 30 08:10 README.md
-rw-rw-rw- 1 yongqiang yongqiang   988 May 30 08:10 setup.py
drwxrwxrwx 1 yongqiang yongqiang   512 May 30 08:10 tensor2tensor/
yongqiang@yongqiang:~/tensor2tensor/v1_0_12/tensor2tensor$

2. Tensorflow 1.4.1

(base) yongqiang@yongqiang:~$ conda env list
# conda environments:
#
base                  *  /home/yongqiang/miniconda3
tf_cpu_1.4.1             /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1

(base) yongqiang@yongqiang:~$
(base) yongqiang@yongqiang:~$ conda activate tf_cpu_1.4.1
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ python
Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 21:41:56)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>>
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
2020-05-30 21:41:06.278325: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
>>> sess.run(hello)
b'Hello, TensorFlow!'
>>> a = tf.constant(10)
>>> b = tf.constant(32)
>>> sess.run(a + b)
42
>>> sess.close()
>>> exit()
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ conda deactivate
(base) yongqiang@yongqiang:~$

3. Walkthrough

Here’s a walkthrough training a good English-to-German translation model using the Transformer model from Attention Is All You Need on WMT data.
https://arxiv.org/abs/1706.03762

Workshop on Statistical Machine Translation，WMT

3.1 安装 tensor2tensor 1.0.12

https://pypi.org/project/tensor2tensor/
Tensor2Tensor, or T2T for short, is a library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research. We keep it running and welcome bug-fixes, but encourage users to use the successor library Trax.

  • 安装 tensor2tensor 1.0.12 - pip install tensor2tensor==1.0.12
(base) yongqiang@yongqiang:~$ conda env list
# conda environments:
#
base                  *  /home/yongqiang/miniconda3
tf_cpu_1.4.1             /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1

(base) yongqiang@yongqiang:~$
(base) yongqiang@yongqiang:~$ conda activate tf_cpu_1.4.1
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ls
miniconda3  pycharm_work  software  tensor2tensor  tensorflow_work  yongqiang
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ pip install tensor2tensor==1.0.12
Collecting tensor2tensor==1.0.12
  Downloading https://files.pythonhosted.org/packages/9c/c3/d14192a122d7beb5f915b11e3f809ee07e636c5aae4259eedaa73dedc7a9/tensor2tensor-1.0.12-py2.py3-none-any.whl (195kB)
    100% |████████████████████████████████| 204kB 9.1kB/s
Requirement already satisfied: numpy in ./miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages (from tensor2tensor==1.0.12) (1.16.4)
Requirement already satisfied: six in ./miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages (from tensor2tensor==1.0.12) (1.15.0)
Collecting sympy (from tensor2tensor==1.0.12)
  Downloading https://files.pythonhosted.org/packages/a4/ea/590e1f2c73a1b8f878a1398b0edbf261d660439a9f851accb39db73d8e2f/sympy-1.6-py3-none-any.whl (5.8MB)
    100% |████████████████████████████████| 5.8MB 88kB/s
Collecting mpmath>=0.19 (from sympy->tensor2tensor==1.0.12)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)",)': /simple/mpmath/
  Downloading https://files.pythonhosted.org/packages/ca/63/3384ebb3b51af9610086b23ea976e6d27d6d97bf140a76a365bd77a3eb32/mpmath-1.1.0.tar.gz (512kB)
    100% |████████████████████████████████| 522kB 151kB/s
Building wheels for collected packages: mpmath
  Running setup.py bdist_wheel for mpmath ... done
  Stored in directory: /home/yongqiang/.cache/pip/wheels/63/9d/8e/37c3f6506ed3f152733a699e92d8e0c9f5e5f01dea262f80ad
Successfully built mpmath
Installing collected packages: mpmath, sympy, tensor2tensor
Successfully installed mpmath-1.1.0 sympy-1.6 tensor2tensor-1.0.12
You are using pip version 10.0.1, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
  • t2t-trainer --registry_help
    See what problems, models, and hyperparameter sets are available. You can easily swap between them (and add new ones).
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ t2t-trainer --registry_help
INFO:tensorflow:
Registry contents:
------------------

  Models: ['attention_lm', 'attention_lm_moe', 'baseline_lstm_seq2seq', 'baseline_lstm_seq2seq_attention', 'blue_net', 'byte_net', 'diagonal_neural_gpu', 'multi_model', 'neural_gpu', 'slice_net', 'transformer', 'xception']

  HParams (by model):
    * attention: ['attention_lm_base', 'attention_lm_moe_base', 'attention_lm_moe_large', 'attention_lm_moe_small']
    * basic: ['basic_1']
    * bluenet: ['bluenet_base', 'bluenet_tiny']
    * bytenet: ['bytenet_base']
    * lstm: ['lstm_attention']
    * multimodel: ['multimodel_base', 'multimodel_tiny']
    * neuralgpu: ['neuralgpu_1']
    * slicenet: ['slicenet_1', 'slicenet_1noam', 'slicenet_1tiny']
    * transformer: ['transformer_base', 'transformer_base_single_gpu', 'transformer_big', 'transformer_big_dr1', 'transformer_big_dr2', 'transformer_big_enfr', 'transformer_big_single_gpu', 'transformer_dr0', 'transformer_dr2', 'transformer_ff1024', 'transformer_ff4096', 'transformer_h1', 'transformer_h16', 'transformer_h32', 'transformer_h4', 'transformer_hs1024', 'transformer_hs256', 'transformer_k128', 'transformer_k256', 'transformer_l2', 'transformer_l4', 'transformer_l8', 'transformer_ls0', 'transformer_ls2', 'transformer_parameter_attention_a', 'transformer_parameter_attention_b', 'transformer_parsing_base', 'transformer_parsing_big', 'transformer_tiny']
    * xception: ['xception_base', 'xception_tiny']

  RangedHParams: ['basic1', 'slicenet1', 'transformer_big_single_gpu']

  Modalities: ['audio:audio_spectral_modality', 'audio:default', 'audio:identity', 'class_label:class_label_2d', 'class_label:default', 'class_label:identity', 'generic:default', 'image:default', 'image:identity', 'image:small_image_modality', 'symbol:default', 'symbol:identity']

(tf_cpu_1.4.1) yongqiang@yongqiang:~$

3.2 创建项目目录

PROBLEM=wmt_ende_tokens_32k
MODEL=transformer
HPARAMS=transformer_base_single_gpu

DATA_DIR=$HOME/t2t_data
TMP_DIR=/tmp/t2t_datagen
TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS

mkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR

TMP_DIR=/tmp/t2t_datagen 修改为 TMP_DIR=$HOME/t2t_datagen 较佳。

PROBLEM=translate_ende_wmt32k 用来生成数据集，并定义任务的类型。这里选择的是翻译任务 (translate)，从英语翻译成德文 (ende)，选择使用 transformer model。

HPARAMS：超参数。
DATA_DIR：用于训练的数据。
TMP_DIR：未经处理的原始数据，tensor2tensor 提供的数据。
TRAIN_DIR：在训练过程中产生的文件。

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ echo $HOME
/home/yongqiang
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll
total 20
drwxr-xr-x 1 yongqiang yongqiang  512 May 31 09:17 ./
drwxr-xr-x 1 root      root       512 Apr 30 17:52 ../
-rw------- 1 yongqiang yongqiang 3816 May 31 10:24 .bash_history
-rw-r--r-- 1 yongqiang yongqiang  220 Apr 30 17:52 .bash_logout
-rw-r--r-- 1 yongqiang yongqiang 4377 May 31 09:17 .bashrc
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:03 .cache/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:53 .conda/
drwx------ 1 yongqiang yongqiang  512 May 31 09:53 .config/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:03 .java/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 20:20 .keras/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:04 .local/
-rw-r--r-- 1 yongqiang yongqiang  807 Apr 30 17:52 .profile
-rw------- 1 yongqiang yongqiang  355 May 30 21:42 .python_history
-rw-r--r-- 1 yongqiang yongqiang    0 May  2 08:15 .sudo_as_admin_successful
-rw------- 1 yongqiang yongqiang  794 May 31 09:17 .viminfo
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:53 miniconda3/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 09:09 pycharm_work/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 09:55 software/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:43 tensor2tensor/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 tensorflow_work/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 yongqiang/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ PROBLEM=wmt_ende_tokens_32k
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ MODEL=transformer
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ HPARAMS=transformer_base_single_gpu
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ DATA_DIR=$HOME/t2t_data
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ TMP_DIR=/tmp/t2t_datagen
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ mkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll
total 20
drwxr-xr-x 1 yongqiang yongqiang  512 May 31 16:28 ./
drwxr-xr-x 1 root      root       512 Apr 30 17:52 ../
-rw------- 1 yongqiang yongqiang 3816 May 31 10:24 .bash_history
-rw-r--r-- 1 yongqiang yongqiang  220 Apr 30 17:52 .bash_logout
-rw-r--r-- 1 yongqiang yongqiang 4377 May 31 09:17 .bashrc
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:03 .cache/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:53 .conda/
drwx------ 1 yongqiang yongqiang  512 May 31 09:53 .config/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:03 .java/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 20:20 .keras/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 00:04 .local/
-rw-r--r-- 1 yongqiang yongqiang  807 Apr 30 17:52 .profile
-rw------- 1 yongqiang yongqiang  355 May 30 21:42 .python_history
-rw-r--r-- 1 yongqiang yongqiang    0 May  2 08:15 .sudo_as_admin_successful
-rw------- 1 yongqiang yongqiang  794 May 31 09:17 .viminfo
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:53 miniconda3/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 09:09 pycharm_work/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 09:55 software/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 16:28 t2t_data/
drwxrwxrwx 1 yongqiang yongqiang  512 May 31 16:28 t2t_train/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:43 tensor2tensor/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 tensorflow_work/
drwxrwxrwx 1 yongqiang yongqiang  512 May 30 07:27 yongqiang/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ls /tmp/t2t_datagen/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

3.3 生成数据

https://github.com/tensorflow/tensor2tensor/tree/v1.0.12

# Generate data
t2t-datagen --data_dir=$DATA_DIR --tmp_dir=$TMP_DIR --num_shards=100 --problem=$PROBLEM

cp $TMP_DIR/tokens.vocab.* $DATA_DIR

https://github.com/tensorflow/tensor2tensor

# Generate data
t2t-datagen --data_dir=$DATA_DIR --tmp_dir=$TMP_DIR --problem=$PROBLEM

https://github.com/tensorflow/tensor2tensor/tree/v1.0.12

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ t2t-datagen --data_dir=$DATA_DIR --tmp_dir=$TMP_DIR --num_shards=100 --problem=$PROBLEM
INFO:tensorflow:Generating problems:
  * wmt_ende_tokens_32k

INFO:tensorflow:Generating training data for wmt_ende_tokens_32k.
INFO:tensorflow:Generating vocab from: [['http://data.statmt.org/wmt16/translation-task/training-parallel-nc-v11.tgz', ['training-parallel-nc-v11/news-commentary-v11.de-en.en', 'training-parallel-nc-v11/news-commentary-v11.de-en.de']], ['http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz', ['commoncrawl.de-en.en', 'commoncrawl.de-en.de', 'commoncrawl.fr-en.en', 'commoncrawl.fr-en.fr']], ['http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz', ['training/europarl-v7.de-en.en', 'training/europarl-v7.de-en.de', 'training/europarl-v7.fr-en.en', 'training/europarl-v7.fr-en.fr']], ['http://www.statmt.org/wmt10/training-giga-fren.tar', ['giga-fren.release2.fixed.en.gz', 'giga-fren.release2.fixed.fr.gz']], ['http://www.statmt.org/wmt13/training-parallel-un.tgz', ['un/undoc.2000.fr-en.en', 'un/undoc.2000.fr-en.fr']]]
INFO:tensorflow:Downloading http://data.statmt.org/wmt16/translation-task/training-parallel-nc-v11.tgz to /tmp/t2t_datagen/training-parallel-nc-v11.tgz
100% completed
INFO:tensorflow:Succesfully downloaded training-parallel-nc-v11.tgz, 75178032 bytes.
INFO:tensorflow:Reading file: training-parallel-nc-v11/news-commentary-v11.de-en.en
INFO:tensorflow:Reading file: training-parallel-nc-v11/news-commentary-v11.de-en.de
INFO:tensorflow:Downloading http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz to /tmp/t2t_datagen/training-parallel-commoncrawl.tgz
100% completed
INFO:tensorflow:Succesfully downloaded training-parallel-commoncrawl.tgz, 918311367 bytes.
INFO:tensorflow:Reading file: commoncrawl.de-en.en
INFO:tensorflow:Reading file: commoncrawl.de-en.de
INFO:tensorflow:Reading file: commoncrawl.fr-en.en
INFO:tensorflow:Reading file: commoncrawl.fr-en.fr
INFO:tensorflow:Downloading http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz to /tmp/t2t_datagen/training-parallel-europarl-v7.tgz
100% completed
INFO:tensorflow:Succesfully downloaded training-parallel-europarl-v7.tgz, 657632379 bytes.
INFO:tensorflow:Reading file: training/europarl-v7.de-en.en
INFO:tensorflow:Reading file: training/europarl-v7.de-en.de
INFO:tensorflow:Reading file: training/europarl-v7.fr-en.en
INFO:tensorflow:Reading file: training/europarl-v7.fr-en.fr
INFO:tensorflow:Downloading http://www.statmt.org/wmt10/training-giga-fren.tar to /tmp/t2t_datagen/training-giga-fren.tar
100% completed
INFO:tensorflow:Succesfully downloaded training-giga-fren.tar, 2595102720 bytes.
INFO:tensorflow:Reading file: giga-fren.release2.fixed.en.gz
INFO:tensorflow:Unpacking subdirectory /tmp/t2t_datagen/giga-fren.release2.fixed.en.gz
INFO:tensorflow:Unpacking /tmp/t2t_datagen/giga-fren.release2.fixed.en.gz to /tmp/t2t_datagen/giga-fren.release2.fixed.en
INFO:tensorflow:Reading file: giga-fren.release2.fixed.fr.gz
INFO:tensorflow:Unpacking subdirectory /tmp/t2t_datagen/giga-fren.release2.fixed.fr.gz
INFO:tensorflow:Unpacking /tmp/t2t_datagen/giga-fren.release2.fixed.fr.gz to /tmp/t2t_datagen/giga-fren.release2.fixed.fr
INFO:tensorflow:Downloading http://www.statmt.org/wmt13/training-parallel-un.tgz to /tmp/t2t_datagen/training-parallel-un.tgz
100% completed
INFO:tensorflow:Succesfully downloaded training-parallel-un.tgz, 2365634246 bytes.
INFO:tensorflow:Reading file: un/undoc.2000.fr-en.en
INFO:tensorflow:Reading file: un/undoc.2000.fr-en.fr
INFO:tensorflow:Trying min_count 500
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 2692
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 1276
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 1387
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 1368
[680, 355, 406, 56, 294, 836, 993, 9, 150, 6, 847, 225, 386, 66, 824, 178, 75, 662, 200, 33, 5]
['This_', 'sen', 'ten', 'ce_', 'was_', 'enc', 'ode', 'd_', 'by_', 'the_', 'Su', 'b', 'wor', 'd', 'Te', 'x', 't', 'En', 'co', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 250
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 4921
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 2180
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 2346
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 2285
[395, 1321, 483, 43, 135, 2261, 2019, 2, 72, 6, 500, 298, 755, 101, 488, 311, 103, 954, 207, 31, 5]
['This_', 'sen', 'ten', 'ce_', 'was_', 'enco', 'ded', '_', 'by_', 'the_', 'Su', 'b', 'wor', 'd', 'Te', 'x', 't', 'En', 'co', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 125
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 9071
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 3656
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 3904
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 3833
[192, 3413, 657, 90, 3537, 1574, 60, 6, 256, 283, 1522, 404, 2153, 945, 455, 29, 4]
['This_', 'sent', 'ence_', 'was_', 'enco', 'ded_', 'by_', 'the_', 'Su', 'b', 'word', 'Te', 'xt', 'En', 'co', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 62
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 16030
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 6094
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 6398
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 6337
[147, 1998, 1162, 62, 85, 1056, 6214, 15, 55, 5, 534, 455, 1287, 271, 563, 3898, 770, 574, 29, 4]
['This_', 'sen', 'ten', 'ce_', 'was_', 'enc', 'ode', 'd_', 'by_', 'the_', 'Su', 'b', 'wor', 'd', 'Te', 'xt', 'En', 'co', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 31
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 26906
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 9865
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 10224
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 10176
[132, 2700, 1926, 75, 4758, 2027, 53, 5, 3728, 2585, 346, 5471, 329, 1095, 7275, 40, 4]
['This_', 'sent', 'ence_', 'was_', 'enco', 'ded_', 'by_', 'the_', 'Sub', 'wor', 'd', 'Tex', 't', 'En', 'cod', 'er_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 15
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 44146
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 15834
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 16294
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 16224
[119, 10073, 565, 71, 6733, 2317, 49, 5, 4054, 3493, 998, 10477, 1899, 6760, 40, 3]
['This_', 'sente', 'nce_', 'was_', 'enco', 'ded_', 'by_', 'the_', 'Sub', 'wor', 'd', 'Text', 'En', 'code', 'r_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 7
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 71625
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 25196
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 25741
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 25660
[113, 12454, 5, 66, 14726, 3859, 46, 4, 5460, 18242, 15158, 1333, 3106, 17159, 54, 3]
['This_', 'sentence', '_', 'was_', 'enco', 'ded_', 'by_', 'the_', 'Sub', 'word', 'Tex', 't', 'En', 'code', 'r_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 3
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 118383
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 40335
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 41029
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 40938
[108, 15212, 62, 24807, 23, 42, 4, 14289, 17583, 26393, 11393, 19634, 95, 3]
['This_', 'sentence_', 'was_', 'encode', 'd_', 'by_', 'the_', 'Sub', 'word', 'Text', 'En', 'code', 'r_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 5
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 87678
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 30452
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 31135
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 31022
[111, 20623, 64, 15493, 4478, 45, 4, 6387, 14372, 12204, 2310, 30424, 23, 3]
['This_', 'sentence_', 'was_', 'enco', 'ded_', 'by_', 'the_', 'Sub', 'word', 'Tex', 't', 'Enco', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Trying min_count 4
INFO:tensorflow:Iteration 0
INFO:tensorflow:vocab_size = 100818
INFO:tensorflow:Iteration 1
INFO:tensorflow:vocab_size = 34547
INFO:tensorflow:Iteration 2
INFO:tensorflow:vocab_size = 35306
INFO:tensorflow:Iteration 3
INFO:tensorflow:vocab_size = 35174
[110, 17462, 63, 30500, 23, 45, 4, 19525, 12792, 11021, 3072, 25323, 21, 3]
['This_', 'sentence_', 'was_', 'encode', 'd_', 'by_', 'the_', 'Sub', 'word', 'Tex', 't', 'Enco', 'der_', '._']
This sentence was encoded by the SubwordTextEncoder.
INFO:tensorflow:Generating case 100000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 200000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 300000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 400000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 500000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 600000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 700000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 800000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 900000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1000000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1100000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1200000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1300000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1400000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1500000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1600000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1700000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1800000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 1900000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2000000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2100000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2200000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2300000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2400000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2500000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2600000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2700000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2800000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 2900000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3000000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3100000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3200000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3300000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3400000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3500000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3600000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3700000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3800000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 3900000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4000000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4100000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4200000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4300000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4400000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating case 4500000 for wmt_ende_tokens_32k-unshuffled-train.
INFO:tensorflow:Generating development data for wmt_ende_tokens_32k.
INFO:tensorflow:Found vocab file: /tmp/t2t_datagen/tokens.vocab.32768
INFO:tensorflow:Downloading http://data.statmt.org/wmt16/translation-task/dev.tgz to /tmp/t2t_datagen/dev.tgz
100% completed
INFO:tensorflow:Succesfully downloaded dev.tgz, 22836484 bytes.
INFO:tensorflow:Shuffling data...
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

--tmp_dir=$TMP_DIR 文件夹下数据：

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /tmp/t2t_datagen/
total 21037120
drwxrwxrwx 1 yongqiang yongqiang        512 Jun  1 03:51 ./
drwxrwxrwt 1 root      root             512 Jun  1 07:42 ../
-rw-r--r-- 1 yongqiang yongqiang    2310368 Jan 12  2013 commoncrawl.cs-en.annotation
-rw-r--r-- 1 yongqiang yongqiang   20745616 Jan 12  2013 commoncrawl.cs-en.cs
-rw-r--r-- 1 yongqiang yongqiang   20655491 Jan 12  2013 commoncrawl.cs-en.en
-rw-r--r-- 1 yongqiang yongqiang   35544656 Jan 12  2013 commoncrawl.de-en.annotation
-rw-r--r-- 1 yongqiang yongqiang  340513464 Jan 12  2013 commoncrawl.de-en.de
-rw-r--r-- 1 yongqiang yongqiang  314178160 Jan 12  2013 commoncrawl.de-en.en
-rw-r--r-- 1 yongqiang yongqiang   25951409 Jan 12  2013 commoncrawl.es-en.annotation
-rw-r--r-- 1 yongqiang yongqiang  248845283 Jan 12  2013 commoncrawl.es-en.en
-rw-r--r-- 1 yongqiang yongqiang  272813663 Jan 12  2013 commoncrawl.es-en.es
-rw-r--r-- 1 yongqiang yongqiang   43896615 Jan 12  2013 commoncrawl.fr-en.annotation
-rw-r--r-- 1 yongqiang yongqiang  434655470 Jan 12  2013 commoncrawl.fr-en.en
-rw-r--r-- 1 yongqiang yongqiang  500374763 Jan 12  2013 commoncrawl.fr-en.fr
-rw-r--r-- 1 yongqiang yongqiang   14945508 Jan 12  2013 commoncrawl.ru-en.annotation
-rw-r--r-- 1 yongqiang yongqiang  116164488 Jan 12  2013 commoncrawl.ru-en.en
-rw-r--r-- 1 yongqiang yongqiang  214585860 Feb 26  2013 commoncrawl.ru-en.ru
drwxr-xr-x 1 yongqiang yongqiang        512 Jan  6  2016 dev/
-rw-rw-rw- 1 yongqiang yongqiang   22836484 Jun  1 03:51 dev.tgz
-rw-rw-rw- 1 yongqiang yongqiang 3789873031 Jun  1 02:22 giga-fren.release2.fixed.en
-rw-rw-r-- 1 yongqiang yongqiang 1214224978 Aug 31  2016 giga-fren.release2.fixed.en.gz
-rw-rw-rw- 1 yongqiang yongqiang 4565271815 Jun  1 02:24 giga-fren.release2.fixed.fr
-rw-rw-r-- 1 yongqiang yongqiang 1380871453 Aug 30  2016 giga-fren.release2.fixed.fr.gz
-rw-rw-rw- 1 yongqiang yongqiang     306558 Jun  1 03:32 tokens.vocab.32768
drwxrwxrwx 1 yongqiang yongqiang        512 Jun  1 00:51 training/
-rw-rw-rw- 1 yongqiang yongqiang 2595102720 Jun  1 02:20 training-giga-fren.tar
-rw-rw-rw- 1 yongqiang yongqiang  918311367 May 31 21:48 training-parallel-commoncrawl.tgz
-rw-rw-rw- 1 yongqiang yongqiang  657632379 Jun  1 00:51 training-parallel-europarl-v7.tgz
drwxrwxr-x 1 yongqiang yongqiang        512 Jan  6  2016 training-parallel-nc-v11/
-rw-rw-rw- 1 yongqiang yongqiang   75178032 May 31 17:39 training-parallel-nc-v11.tgz
-rw-rw-rw- 1 yongqiang yongqiang 2365634246 Jun  1 03:28 training-parallel-un.tgz
drwxrwxrwx 1 yongqiang yongqiang        512 Jun  1 03:30 un/
-rw-rw-rw- 1 yongqiang yongqiang     332974 Jun  1 03:51 wmt_ende_tok_dev.lang1
-rw-rw-rw- 1 yongqiang yongqiang     385433 Jun  1 03:51 wmt_ende_tok_dev.lang2
-rw-rw-rw- 1 yongqiang yongqiang  636460447 Jun  1 03:33 wmt_ende_tok_train.lang1
-rw-rw-rw- 1 yongqiang yongqiang  710260525 Jun  1 03:34 wmt_ende_tok_train.lang2
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

其中 .lang1 是英语语料，要翻译的数据。.lang2 是德语语料，翻译后的数据。

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /tmp/t2t_datagen/un
total 8287744
drwxrwxrwx 1 yongqiang yongqiang        512 Jun  1 03:30 ./
drwxrwxrwx 1 yongqiang yongqiang        512 Jun  1 03:51 ../
-rw-r--r-- 1 yongqiang yongqiang 1851755840 Oct 13  2011 undoc.2000.es-en.en
-rw-r--r-- 1 yongqiang yongqiang 2121526762 Oct 13  2011 undoc.2000.es-en.es
-rw-r--r-- 1 yongqiang yongqiang 2085411017 Oct 13  2011 undoc.2000.fr-en.en
-rw-r--r-- 1 yongqiang yongqiang 2427161501 Oct 13  2011 undoc.2000.fr-en.fr
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /tmp/t2t_datagen/dev/
total 83520
drwxr-xr-x 1 yongqiang yongqiang    512 Jan  6  2016 ./
drwxrwxrwx 1 yongqiang yongqiang    512 Jun  1 03:51 ../
-rw-r--r-- 1 yongqiang yongqiang  16384 Jan 16  2014 .newsdev2014-ref.en.sgm.swp
-rw-r--r-- 1 yongqiang yongqiang  16384 Jan 16  2014 .newstest2013-ref.en.sgm.swp
-rw-r--r-- 1 yongqiang yongqiang 326194 Dec  3  2009 news-test2008-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 360808 Dec  3  2009 news-test2008-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 325852 Dec  3  2009 news-test2008-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 351112 Dec  3  2009 news-test2008-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 351196 Dec  3  2009 news-test2008-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 349559 Dec  3  2009 news-test2008-ref.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang 325101 Dec  3  2009 news-test2008-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 359715 Dec  3  2009 news-test2008-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 324759 Dec  3  2009 news-test2008-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 350019 Dec  3  2009 news-test2008-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 350103 Dec  3  2009 news-test2008-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 348466 Dec  3  2009 news-test2008-src.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang 267659 Dec 23  2009 news-test2008.cs
-rw-r--r-- 1 yongqiang yongqiang 298369 Dec 23  2009 news-test2008.de
-rw-r--r-- 1 yongqiang yongqiang 263240 Dec 23  2009 news-test2008.en
-rw-r--r-- 1 yongqiang yongqiang 288487 Dec 23  2009 news-test2008.es
-rw-r--r-- 1 yongqiang yongqiang 301220 Dec 23  2009 news-test2008.fr
-rw-r--r-- 1 yongqiang yongqiang  68931 Jan 16  2014 newsdev2014-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 137001 Jan 16  2014 newsdev2014-ref.hi.sgm
-rw-r--r-- 1 yongqiang yongqiang  68918 Jan 13  2014 newsdev2014-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 136988 Jan 13  2014 newsdev2014-src.hi.sgm
-rw-r--r-- 1 yongqiang yongqiang  55133 Jan 13  2014 newsdev2014.en
-rw-r--r-- 1 yongqiang yongqiang 123204 Jan 13  2014 newsdev2014.hi
-rw-r--r-- 1 yongqiang yongqiang 216726 Jan 28  2015 newsdev2015-enfi-ref.fi.sgm
-rw-r--r-- 1 yongqiang yongqiang 207286 Jan 28  2015 newsdev2015-enfi-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 207299 Jan 28  2015 newsdev2015-fien-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 216713 Jan 28  2015 newsdev2015-fien-src.fi.sgm
-rw-rw-r-- 1 yongqiang yongqiang 336460 Jan  6  2016 newsdev2016-enro-ref.ro.sgm
-rw-rw-r-- 1 yongqiang yongqiang 306002 Jan  6  2016 newsdev2016-enro-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 139478 Jan  6  2016 newsdev2016-entr-ref.tr.sgm
-rw-rw-r-- 1 yongqiang yongqiang 135181 Jan  6  2016 newsdev2016-entr-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 306015 Jan  6  2016 newsdev2016-roen-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 336447 Jan  6  2016 newsdev2016-roen-src.ro.sgm
-rw-rw-r-- 1 yongqiang yongqiang 135194 Jan  6  2016 newsdev2016-tren-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 139465 Jan  6  2016 newsdev2016-tren-src.tr.sgm
-rw-r--r-- 1 yongqiang yongqiang 188019 Jan 28  2015 newsdiscussdev2015-enfr-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 171890 Jan 28  2015 newsdiscussdev2015-enfr-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 171903 Jan 28  2015 newsdiscussdev2015-fren-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 188006 Jan 28  2015 newsdiscussdev2015-fren-src.fr.sgm
-rw-rw-r-- 1 yongqiang yongqiang 188490 Jan  6  2016 newsdiscusstest2015-enfr-ref.fr.sgm
-rw-rw-r-- 1 yongqiang yongqiang 170340 Jan  6  2016 newsdiscusstest2015-enfr-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 170353 Jan  6  2016 newsdiscusstest2015-fren-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 188477 Jan  6  2016 newsdiscusstest2015-fren-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang  77096 Dec  3  2009 newssyscomb2009-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang  83899 Dec  3  2009 newssyscomb2009-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang  76771 Dec  3  2009 newssyscomb2009-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang  83167 Dec  3  2009 newssyscomb2009-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang  84788 Dec  3  2009 newssyscomb2009-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang  81826 Dec  3  2009 newssyscomb2009-ref.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang  80940 Dec  3  2009 newssyscomb2009-ref.it.sgm
-rw-r--r-- 1 yongqiang yongqiang  76783 Dec  3  2009 newssyscomb2009-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang  83587 Dec  3  2009 newssyscomb2009-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang  76458 Dec  3  2009 newssyscomb2009-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang  82856 Dec  3  2009 newssyscomb2009-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang  84482 Dec  3  2009 newssyscomb2009-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang  81513 Dec  3  2009 newssyscomb2009-src.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang  80654 Dec  3  2009 newssyscomb2009-src.it.sgm
-rw-r--r-- 1 yongqiang yongqiang  62880 Dec 10  2011 newssyscomb2009.cs
-rw-r--r-- 1 yongqiang yongqiang  69689 Dec 10  2011 newssyscomb2009.de
-rw-r--r-- 1 yongqiang yongqiang  62557 Dec 10  2011 newssyscomb2009.en
-rw-r--r-- 1 yongqiang yongqiang  68947 Dec 10  2011 newssyscomb2009.es
-rw-r--r-- 1 yongqiang yongqiang  70529 Dec 10  2011 newssyscomb2009.fr
-rw-r--r-- 1 yongqiang yongqiang 417868 Dec  3  2009 newstest2009-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 452329 Dec  3  2009 newstest2009-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 415276 Dec  3  2009 newstest2009-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 443880 Dec  3  2009 newstest2009-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 460824 Dec  3  2009 newstest2009-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 448263 Dec  3  2009 newstest2009-ref.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang 434489 Dec  3  2009 newstest2009-ref.it.sgm
-rw-r--r-- 1 yongqiang yongqiang 416524 Dec  3  2009 newstest2009-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 450989 Dec  3  2009 newstest2009-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 413934 Dec  3  2009 newstest2009-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 442554 Dec  3  2009 newstest2009-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 459552 Dec  3  2009 newstest2009-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 446918 Dec  3  2009 newstest2009-src.hu.sgm
-rw-r--r-- 1 yongqiang yongqiang 433226 Dec  3  2009 newstest2009-src.it.sgm
-rw-r--r-- 1 yongqiang yongqiang 413934 Dec  3  2009 newstest2009-src.xx.sgm
-rw-r--r-- 1 yongqiang yongqiang 348154 Dec 23  2009 newstest2009.cs
-rw-r--r-- 1 yongqiang yongqiang 382617 Dec 23  2009 newstest2009.de
-rw-r--r-- 1 yongqiang yongqiang 345562 Dec 23  2009 newstest2009.en
-rw-r--r-- 1 yongqiang yongqiang 374178 Dec 23  2009 newstest2009.es
-rw-r--r-- 1 yongqiang yongqiang 391108 Dec 23  2009 newstest2009.fr
-rw-r--r-- 1 yongqiang yongqiang 406600 Mar  5  2010 newstest2010-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 453580 Mar  5  2010 newstest2010-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 398937 Mar  5  2010 newstest2010-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 434326 Mar  5  2010 newstest2010-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 453687 Mar  5  2010 newstest2010-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 405159 Mar  5  2010 newstest2010-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 452139 Mar  5  2010 newstest2010-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 397496 Mar  5  2010 newstest2010-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 432885 Mar  5  2010 newstest2010-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 452246 Mar  5  2010 newstest2010-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 336311 Mar  5  2010 newstest2010.cs
-rw-r--r-- 1 yongqiang yongqiang 383291 Mar  5  2010 newstest2010.de
-rw-r--r-- 1 yongqiang yongqiang 328648 Mar  5  2010 newstest2010.en
-rw-r--r-- 1 yongqiang yongqiang 364037 Mar  5  2010 newstest2010.es
-rw-r--r-- 1 yongqiang yongqiang 383398 Mar  5  2010 newstest2010.fr
-rw-r--r-- 1 yongqiang yongqiang 485879 Nov 24  2011 newstest2011-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 530959 Mar 16  2011 newstest2011-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 477272 Mar 16  2011 newstest2011-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 514062 Mar 16  2011 newstest2011-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 537007 Mar 16  2011 newstest2011-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 484546 Mar 16  2011 newstest2011-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 529626 Mar 16  2011 newstest2011-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 475939 Mar 16  2011 newstest2011-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 512729 Mar 16  2011 newstest2011-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 535674 Mar 16  2011 newstest2011-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 405486 Dec 10  2011 newstest2011.cs
-rw-r--r-- 1 yongqiang yongqiang 450571 Dec 10  2011 newstest2011.de
-rw-r--r-- 1 yongqiang yongqiang 396884 Dec 10  2011 newstest2011.en
-rw-r--r-- 1 yongqiang yongqiang 433659 Dec 10  2011 newstest2011.es
-rw-r--r-- 1 yongqiang yongqiang 456618 Dec 10  2011 newstest2011.fr
-rw-r--r-- 1 yongqiang yongqiang 484553 Feb 16  2012 newstest2012-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 524750 Feb 16  2012 newstest2012-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 459733 Mar  3  2012 newstest2012-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 504934 Feb 16  2012 newstest2012-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 518968 Feb 16  2012 newstest2012-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 776643 Jan 23  2013 newstest2012-ref.ru.sgm
-rw-r--r-- 1 yongqiang yongqiang 483352 Mar  3  2012 newstest2012-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 523549 Mar  3  2012 newstest2012-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 458532 Mar  3  2012 newstest2012-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 503733 Mar  3  2012 newstest2012-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 517767 Mar  3  2012 newstest2012-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 776618 Jan 23  2013 newstest2012-src.ru.sgm
-rw-r--r-- 1 yongqiang yongqiang 406578 Jan 23  2013 newstest2012.cs
-rw-r--r-- 1 yongqiang yongqiang 446765 Jan 23  2013 newstest2012.de
-rw-r--r-- 1 yongqiang yongqiang 381734 Jan 23  2013 newstest2012.en
-rw-r--r-- 1 yongqiang yongqiang 426936 Jan 23  2013 newstest2012.es
-rw-r--r-- 1 yongqiang yongqiang 440985 Jan 23  2013 newstest2012.fr
-rw-r--r-- 1 yongqiang yongqiang 699810 Jan 23  2013 newstest2012.ru
-rw-r--r-- 1 yongqiang yongqiang 423686 Dec 13  2013 newstest2013-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 457608 Dec 13  2013 newstest2013-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 405170 Dec 13  2013 newstest2013-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 451575 Dec 13  2013 newstest2013-ref.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 465895 Dec 13  2013 newstest2013-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 699161 Dec 13  2013 newstest2013-ref.ru.sgm
-rw-r--r-- 1 yongqiang yongqiang 423049 Dec 13  2013 newstest2013-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 456971 Dec 13  2013 newstest2013-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 404533 Dec 13  2013 newstest2013-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 450938 Dec 13  2013 newstest2013-src.es.sgm
-rw-r--r-- 1 yongqiang yongqiang 465258 Dec 13  2013 newstest2013-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 698524 Dec 13  2013 newstest2013-src.ru.sgm
-rw-r--r-- 1 yongqiang yongqiang 351491 Dec 13  2013 newstest2013.cs
-rw-r--r-- 1 yongqiang yongqiang 385433 Dec 13  2013 newstest2013.de
-rw-r--r-- 1 yongqiang yongqiang 332974 Dec 13  2013 newstest2013.en
-rw-r--r-- 1 yongqiang yongqiang 379357 Dec 13  2013 newstest2013.es
-rw-r--r-- 1 yongqiang yongqiang 393465 Dec 13  2013 newstest2013.fr
-rw-r--r-- 1 yongqiang yongqiang 626964 Dec 13  2013 newstest2013.ru
-rw-r--r-- 1 yongqiang yongqiang 444462 Jul  1  2014 newstest2014-csen-ref.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 428063 Jul  1  2014 newstest2014-csen-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 442757 Jul  1  2014 newstest2014-csen-src.cs.sgm
-rw-r--r-- 1 yongqiang yongqiang 426358 Jul  1  2014 newstest2014-csen-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 468154 Jul  1  2014 newstest2014-deen-ref.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 428650 Jul  1  2014 newstest2014-deen-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 466173 Jul  1  2014 newstest2014-deen-src.de.sgm
-rw-r--r-- 1 yongqiang yongqiang 426669 Jul  1  2014 newstest2014-deen-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 443970 Jul  1  2014 newstest2014-fren-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 514587 Jul  1  2014 newstest2014-fren-ref.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 441845 Jul  1  2014 newstest2014-fren-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 512462 Jul  1  2014 newstest2014-fren-src.fr.sgm
-rw-r--r-- 1 yongqiang yongqiang 356077 Jul  1  2014 newstest2014-hien-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 820215 Jul  1  2014 newstest2014-hien-ref.hi.sgm
-rw-r--r-- 1 yongqiang yongqiang 353792 Jul  1  2014 newstest2014-hien-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 818658 Jul  1  2014 newstest2014-hien-src.hi.sgm
-rw-r--r-- 1 yongqiang yongqiang 443149 Jul  1  2014 newstest2014-ruen-ref.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 763821 Jul  1  2014 newstest2014-ruen-ref.ru.sgm
-rw-r--r-- 1 yongqiang yongqiang 441036 Jul  1  2014 newstest2014-ruen-src.en.sgm
-rw-r--r-- 1 yongqiang yongqiang 761708 Jul  1  2014 newstest2014-ruen-src.ru.sgm
-rw-rw-r-- 1 yongqiang yongqiang 333884 Jan  6  2016 newstest2015-csen-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 337118 Jan  6  2016 newstest2015-csen-src.cs.sgm
-rw-rw-r-- 1 yongqiang yongqiang 288866 Jan  6  2016 newstest2015-deen-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 316788 Jan  6  2016 newstest2015-deen-src.de.sgm
-rw-rw-r-- 1 yongqiang yongqiang 337131 Jan  6  2016 newstest2015-encs-ref.cs.sgm
-rw-rw-r-- 1 yongqiang yongqiang 333871 Jan  6  2016 newstest2015-encs-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 316801 Jan  6  2016 newstest2015-ende-ref.de.sgm
-rw-rw-r-- 1 yongqiang yongqiang 288853 Jan  6  2016 newstest2015-ende-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 185637 Jan  6  2016 newstest2015-enfi-ref.fi.sgm
-rw-rw-r-- 1 yongqiang yongqiang 178431 Jan  6  2016 newstest2015-enfi-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 658969 Jan  6  2016 newstest2015-enru-ref.ru.sgm
-rw-rw-r-- 1 yongqiang yongqiang 406258 Jan  6  2016 newstest2015-enru-src.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 178444 Jan  6  2016 newstest2015-fien-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 185624 Jan  6  2016 newstest2015-fien-src.fi.sgm
-rw-rw-r-- 1 yongqiang yongqiang 406271 Jan  6  2016 newstest2015-ruen-ref.en.sgm
-rw-rw-r-- 1 yongqiang yongqiang 658956 Jan  6  2016 newstest2015-ruen-src.ru.sgm
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /tmp/t2t_datagen/training/
total 2028544
drwxrwxrwx 1 yongqiang yongqiang       512 Jun  1 00:51 ./
drwxrwxrwx 1 yongqiang yongqiang       512 Jun  1 03:51 ../
-rw-r--r-- 1 yongqiang yongqiang  98150314 Nov 21  2011 europarl-v7.cs-en.cs
-rw-r--r-- 1 yongqiang yongqiang  94299411 Nov 21  2011 europarl-v7.cs-en.en
-rw-r--r-- 1 yongqiang yongqiang 328463491 Nov 21  2011 europarl-v7.de-en.de
-rw-r--r-- 1 yongqiang yongqiang 287250069 Nov 21  2011 europarl-v7.de-en.en
-rw-r--r-- 1 yongqiang yongqiang 294931656 Nov 21  2011 europarl-v7.es-en.en
-rw-r--r-- 1 yongqiang yongqiang 324915736 Nov 21  2011 europarl-v7.es-en.es
-rw-r--r-- 1 yongqiang yongqiang 301523301 Nov 21  2011 europarl-v7.fr-en.en
-rw-r--r-- 1 yongqiang yongqiang 346919801 Nov 21  2011 europarl-v7.fr-en.fr
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

training-parallel-nc-v11 是一个词典，运行的过程会先检查 training-parallel-nc-v11.tgz 是否存在。如果不存在，则下载。如果存在，则直接使用 training-parallel-nc-v11 文件夹中的词典。如果想使用自己的词典，可以直接替换文件夹里面的内容，或者是自定义一个 problem。

TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS 文件夹下数据：

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /tmp/t2t_datagen/training/
total 2028544
drwxrwxrwx 1 yongqiang yongqiang       512 Jun  1 00:51 ./
drwxrwxrwx 1 yongqiang yongqiang       512 Jun  1 03:51 ../
-rw-r--r-- 1 yongqiang yongqiang  98150314 Nov 21  2011 europarl-v7.cs-en.cs
-rw-r--r-- 1 yongqiang yongqiang  94299411 Nov 21  2011 europarl-v7.cs-en.en
-rw-r--r-- 1 yongqiang yongqiang 328463491 Nov 21  2011 europarl-v7.de-en.de
-rw-r--r-- 1 yongqiang yongqiang 287250069 Nov 21  2011 europarl-v7.de-en.en
-rw-r--r-- 1 yongqiang yongqiang 294931656 Nov 21  2011 europarl-v7.es-en.en
-rw-r--r-- 1 yongqiang yongqiang 324915736 Nov 21  2011 europarl-v7.es-en.es
-rw-r--r-- 1 yongqiang yongqiang 301523301 Nov 21  2011 europarl-v7.fr-en.en
-rw-r--r-- 1 yongqiang yongqiang 346919801 Nov 21  2011 europarl-v7.fr-en.fr
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxr-xr-x 1 yongqiang yongqiang 512 May 31 16:28 ../
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 wmt_ende_tokens_32k/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/wmt_ende_tokens_32k/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ../
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 transformer-transformer_base_single_gpu/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ../
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

100 个 train 文件和 1 个 dev 文件，都是 TFRecord 格式的，在训练过程中使用，同时还会生成一个字词表。

(tf_cpu_1.4.1) yongqiang@yongqiang:~$ cp $TMP_DIR/tokens.vocab.* $DATA_DIR
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ../
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/wmt_ende_tokens_32k/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ../
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 transformer-transformer_base_single_gpu/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_train/
total 0
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 ./
drwxr-xr-x 1 yongqiang yongqiang 512 May 31 16:28 ../
drwxrwxrwx 1 yongqiang yongqiang 512 May 31 16:28 wmt_ende_tokens_32k/
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ ll /home/yongqiang/t2t_data/
total 753680
drwxrwxrwx 1 yongqiang yongqiang     512 Jun  1 07:51 ./
drwxr-xr-x 1 yongqiang yongqiang     512 May 31 16:28 ../
-rw-rw-rw- 1 yongqiang yongqiang  306558 Jun  1 07:51 tokens.vocab.32768
-rw-rw-rw- 1 yongqiang yongqiang  452958 Jun  1 03:52 wmt_ende_tokens_32k-dev-00000-of-00001
-rw-rw-rw- 1 yongqiang yongqiang 7667929 Jun  1 03:51 wmt_ende_tokens_32k-train-00000-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7640828 Jun  1 03:51 wmt_ende_tokens_32k-train-00001-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7674237 Jun  1 03:51 wmt_ende_tokens_32k-train-00002-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7700999 Jun  1 03:51 wmt_ende_tokens_32k-train-00003-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7663250 Jun  1 03:51 wmt_ende_tokens_32k-train-00004-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7652482 Jun  1 03:51 wmt_ende_tokens_32k-train-00005-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7642697 Jun  1 03:51 wmt_ende_tokens_32k-train-00006-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7676712 Jun  1 03:51 wmt_ende_tokens_32k-train-00007-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7639986 Jun  1 03:51 wmt_ende_tokens_32k-train-00008-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7640001 Jun  1 03:51 wmt_ende_tokens_32k-train-00009-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7673384 Jun  1 03:51 wmt_ende_tokens_32k-train-00010-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7683955 Jun  1 03:51 wmt_ende_tokens_32k-train-00011-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7647624 Jun  1 03:51 wmt_ende_tokens_32k-train-00012-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7687257 Jun  1 03:51 wmt_ende_tokens_32k-train-00013-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7688525 Jun  1 03:51 wmt_ende_tokens_32k-train-00014-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7665535 Jun  1 03:51 wmt_ende_tokens_32k-train-00015-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7690883 Jun  1 03:51 wmt_ende_tokens_32k-train-00016-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7663359 Jun  1 03:51 wmt_ende_tokens_32k-train-00017-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7686219 Jun  1 03:51 wmt_ende_tokens_32k-train-00018-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7648707 Jun  1 03:51 wmt_ende_tokens_32k-train-00019-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7718829 Jun  1 03:51 wmt_ende_tokens_32k-train-00020-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7686182 Jun  1 03:51 wmt_ende_tokens_32k-train-00021-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7675800 Jun  1 03:51 wmt_ende_tokens_32k-train-00022-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7701203 Jun  1 03:51 wmt_ende_tokens_32k-train-00023-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7657998 Jun  1 03:51 wmt_ende_tokens_32k-train-00024-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7670140 Jun  1 03:51 wmt_ende_tokens_32k-train-00025-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7675587 Jun  1 03:51 wmt_ende_tokens_32k-train-00026-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7654837 Jun  1 03:51 wmt_ende_tokens_32k-train-00027-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7677641 Jun  1 03:51 wmt_ende_tokens_32k-train-00028-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7670186 Jun  1 03:51 wmt_ende_tokens_32k-train-00029-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7667826 Jun  1 03:51 wmt_ende_tokens_32k-train-00030-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7683834 Jun  1 03:51 wmt_ende_tokens_32k-train-00031-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7690318 Jun  1 03:51 wmt_ende_tokens_32k-train-00032-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7675815 Jun  1 03:51 wmt_ende_tokens_32k-train-00033-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7634405 Jun  1 03:51 wmt_ende_tokens_32k-train-00034-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7674313 Jun  1 03:51 wmt_ende_tokens_32k-train-00035-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7668282 Jun  1 03:51 wmt_ende_tokens_32k-train-00036-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7656065 Jun  1 03:51 wmt_ende_tokens_32k-train-00037-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7643719 Jun  1 03:51 wmt_ende_tokens_32k-train-00038-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7664325 Jun  1 03:51 wmt_ende_tokens_32k-train-00039-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7647871 Jun  1 03:51 wmt_ende_tokens_32k-train-00040-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7616733 Jun  1 03:51 wmt_ende_tokens_32k-train-00041-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7637022 Jun  1 03:51 wmt_ende_tokens_32k-train-00042-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7664961 Jun  1 03:51 wmt_ende_tokens_32k-train-00043-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7645109 Jun  1 03:51 wmt_ende_tokens_32k-train-00044-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7679298 Jun  1 03:51 wmt_ende_tokens_32k-train-00045-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7655371 Jun  1 03:51 wmt_ende_tokens_32k-train-00046-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7645346 Jun  1 03:51 wmt_ende_tokens_32k-train-00047-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7660534 Jun  1 03:51 wmt_ende_tokens_32k-train-00048-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7677668 Jun  1 03:51 wmt_ende_tokens_32k-train-00049-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7656197 Jun  1 03:51 wmt_ende_tokens_32k-train-00050-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7695677 Jun  1 03:51 wmt_ende_tokens_32k-train-00051-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7681680 Jun  1 03:51 wmt_ende_tokens_32k-train-00052-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7677127 Jun  1 03:51 wmt_ende_tokens_32k-train-00053-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7658826 Jun  1 03:51 wmt_ende_tokens_32k-train-00054-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7651920 Jun  1 03:51 wmt_ende_tokens_32k-train-00055-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7661295 Jun  1 03:51 wmt_ende_tokens_32k-train-00056-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7664402 Jun  1 03:51 wmt_ende_tokens_32k-train-00057-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7664979 Jun  1 03:51 wmt_ende_tokens_32k-train-00058-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7651649 Jun  1 03:51 wmt_ende_tokens_32k-train-00059-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7708872 Jun  1 03:51 wmt_ende_tokens_32k-train-00060-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7654428 Jun  1 03:51 wmt_ende_tokens_32k-train-00061-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7648101 Jun  1 03:51 wmt_ende_tokens_32k-train-00062-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7684443 Jun  1 03:51 wmt_ende_tokens_32k-train-00063-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7681919 Jun  1 03:51 wmt_ende_tokens_32k-train-00064-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7658840 Jun  1 03:52 wmt_ende_tokens_32k-train-00065-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7674366 Jun  1 03:52 wmt_ende_tokens_32k-train-00066-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7644321 Jun  1 03:52 wmt_ende_tokens_32k-train-00067-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7664720 Jun  1 03:52 wmt_ende_tokens_32k-train-00068-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7684936 Jun  1 03:52 wmt_ende_tokens_32k-train-00069-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7673587 Jun  1 03:52 wmt_ende_tokens_32k-train-00070-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7612716 Jun  1 03:52 wmt_ende_tokens_32k-train-00071-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7673989 Jun  1 03:52 wmt_ende_tokens_32k-train-00072-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7679255 Jun  1 03:52 wmt_ende_tokens_32k-train-00073-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7658291 Jun  1 03:52 wmt_ende_tokens_32k-train-00074-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7633770 Jun  1 03:52 wmt_ende_tokens_32k-train-00075-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7660341 Jun  1 03:52 wmt_ende_tokens_32k-train-00076-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7672302 Jun  1 03:52 wmt_ende_tokens_32k-train-00077-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7668570 Jun  1 03:52 wmt_ende_tokens_32k-train-00078-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7640331 Jun  1 03:52 wmt_ende_tokens_32k-train-00079-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7679929 Jun  1 03:52 wmt_ende_tokens_32k-train-00080-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7670161 Jun  1 03:52 wmt_ende_tokens_32k-train-00081-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7629498 Jun  1 03:52 wmt_ende_tokens_32k-train-00082-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7667861 Jun  1 03:52 wmt_ende_tokens_32k-train-00083-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7633190 Jun  1 03:52 wmt_ende_tokens_32k-train-00084-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7668988 Jun  1 03:52 wmt_ende_tokens_32k-train-00085-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7715339 Jun  1 03:52 wmt_ende_tokens_32k-train-00086-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7691430 Jun  1 03:52 wmt_ende_tokens_32k-train-00087-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7693875 Jun  1 03:52 wmt_ende_tokens_32k-train-00088-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7688168 Jun  1 03:52 wmt_ende_tokens_32k-train-00089-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7680385 Jun  1 03:52 wmt_ende_tokens_32k-train-00090-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7669305 Jun  1 03:52 wmt_ende_tokens_32k-train-00091-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7673107 Jun  1 03:52 wmt_ende_tokens_32k-train-00092-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7691164 Jun  1 03:52 wmt_ende_tokens_32k-train-00093-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7735837 Jun  1 03:52 wmt_ende_tokens_32k-train-00094-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7654446 Jun  1 03:52 wmt_ende_tokens_32k-train-00095-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7679692 Jun  1 03:52 wmt_ende_tokens_32k-train-00096-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7668198 Jun  1 03:52 wmt_ende_tokens_32k-train-00097-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7650696 Jun  1 03:52 wmt_ende_tokens_32k-train-00098-of-00100
-rw-rw-rw- 1 yongqiang yongqiang 7653008 Jun  1 03:52 wmt_ende_tokens_32k-train-00099-of-00100
(tf_cpu_1.4.1) yongqiang@yongqiang:~$

3.4 训练阶段

https://github.com/tensorflow/tensor2tensor/tree/v1.0.12

# Train
# If you run out of memory, add --hparams='batch_size=2048' or even 1024.
t2t-trainer --data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR

https://github.com/tensorflow/tensor2tensor

# Train
# If you run out of memory, add --hparams='batch_size=1024'.
t2t-trainer --data_dir=$DATA_DIR --problem=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR

注意 problem 和 problems 的区别，否则会报 NoneType 的错误。

  • Segmentation fault (core dumped)
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ t2t-trainer --data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR
INFO:tensorflow:Creating experiment, storing model files in /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Using config: {'_save_checkpoints_steps': None, '_task_id': 0, '_save_summary_steps': 100, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f1a0850cef0>, '_num_ps_replicas': 0, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_worker_replicas': 0, '_save_checkpoints_secs': 600, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_keep_checkpoint_max': 20, '_is_chief': True, '_master': '', '_environment': 'local', '_keep_checkpoint_every_n_hours': 10000, '_tf_random_seed': None, '_model_dir': '/home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu', '_task_type': None, '_session_config': allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
}
INFO:tensorflow:Performing local training.
WARNING:tensorflow:From /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Doing model_fn_body took 1.564 sec.
INFO:tensorflow:This model_fn took 1.691 sec.
WARNING:tensorflow:From /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages/tensor2tensor/utils/trainer_utils.py:317: get_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_global_step
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/target_space_embedding/kernel                                                    shape    (32, 512)              size    16384
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_0                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_10                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_11                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_12                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_13                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_14                                           shape    (1938, 512)            size    992256
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_15                                           shape    (1938, 512)            size    992256
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_1                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_2                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_3                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_4                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_5                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_6                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_7                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_8                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_9                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Total trainable variables size: 60038144
INFO:tensorflow:Computing gradients for global model_fn.
INFO:tensorflow:Global model_fn finished.
INFO:tensorflow:Create CheckpointSaverHook.
2020-06-01 07:55:08.844229: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Segmentation fault (core dumped)
(tf_cpu_1.4.1) yongqiang@yongqiang:~$
  • t2t-trainer --data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR --hparams='batch_size=1024'
(tf_cpu_1.4.1) yongqiang@yongqiang:~$ t2t-trainer --data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR --hparams='batch_size=1024'
INFO:tensorflow:Creating experiment, storing model files in /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Using config: {'_log_step_count_steps': 100, '_keep_checkpoint_max': 20, '_save_checkpoints_steps': None, '_environment': 'local', '_save_checkpoints_secs': 600, '_task_type': None, '_tf_random_seed': None, '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_session_config': allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe1d260cef0>, '_model_dir': '/home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu', '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
}
INFO:tensorflow:Performing local training.
WARNING:tensorflow:From /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Doing model_fn_body took 1.528 sec.
INFO:tensorflow:This model_fn took 1.657 sec.
WARNING:tensorflow:From /home/yongqiang/miniconda3/envs/tf_cpu_1.4.1/lib/python3.5/site-packages/tensor2tensor/utils/trainer_utils.py:317: get_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_global_step
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_0/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_0/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_1/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_1/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_2/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_2/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_3/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_3/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_4/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_4/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/decoder/layer_5/decoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/kv_transform_single/bias                        shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/kv_transform_single/kernel                      shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/output_transform_single/bias                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/output_transform_single/kernel                  shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/q_transform_single/bias                         shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/encdec_attention/q_transform_single/kernel                       shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_2/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/decoder/layer_5/layer_norm_2/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_0/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_0/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_1/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_1/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_2/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_2/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_3/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_3/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_4/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_4/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/qkv_transform_single/bias                 shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/encoder/layer_5/encoder_self_attention/qkv_transform_single/kernel               shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm_1/layer_norm_bias                                     shape    (512,)                 size    512
INFO:tensorflow:Weight    body/encoder/layer_5/layer_norm_1/layer_norm_scale                                    shape    (512,)                 size    512
INFO:tensorflow:Weight    body/target_space_embedding/kernel                                                    shape    (32, 512)              size    16384
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_0                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_10                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_11                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_12                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_13                                           shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_14                                           shape    (1938, 512)            size    992256
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_15                                           shape    (1938, 512)            size    992256
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_1                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_2                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_3                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_4                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_5                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_6                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_7                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_8                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Weight    symbol_modality_31022_512/shared/weights_9                                            shape    (1939, 512)            size    992768
INFO:tensorflow:Total trainable variables size: 60038144
INFO:tensorflow:Computing gradients for global model_fn.
INFO:tensorflow:Global model_fn finished.
INFO:tensorflow:Create CheckpointSaverHook.
2020-06-01 07:59:36.595618: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
INFO:tensorflow:Saving checkpoints for 1 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:loss = 9.617761, step = 1
INFO:tensorflow:global_step/sec: 0.184352
INFO:tensorflow:loss = 8.638795, step = 101 (542.447 sec)
INFO:tensorflow:Saving checkpoints for 111 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.182636
INFO:tensorflow:loss = 8.1433325, step = 201 (547.531 sec)
INFO:tensorflow:Saving checkpoints for 221 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.184702
INFO:tensorflow:loss = 7.774977, step = 301 (541.413 sec)
INFO:tensorflow:Saving checkpoints for 332 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.183971
INFO:tensorflow:loss = 7.6570582, step = 401 (543.564 sec)
INFO:tensorflow:Saving checkpoints for 444 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.183122
INFO:tensorflow:loss = 7.5161967, step = 501 (546.084 sec)
INFO:tensorflow:Saving checkpoints for 555 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.185353
INFO:tensorflow:loss = 7.1285443, step = 601 (539.511 sec)
INFO:tensorflow:Saving checkpoints for 665 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.182844
INFO:tensorflow:loss = 6.6412644, step = 701 (546.914 sec)
INFO:tensorflow:Saving checkpoints for 776 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.183772
INFO:tensorflow:loss = 6.9026318, step = 801 (544.153 sec)
INFO:tensorflow:Saving checkpoints for 887 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.183551
INFO:tensorflow:loss = 6.866945, step = 901 (544.809 sec)
INFO:tensorflow:Saving checkpoints for 997 into /home/yongqiang/t2t_train/wmt_ende_tokens_32k/transformer-transformer_base_single_gpu/model.ckpt.
INFO:tensorflow:global_step/sec: 0.182267
INFO:tensorflow:loss = 6.583713, step = 1001 (548.645 sec)
INFO:tensorflow:global_step/sec: 0.187032
......

3.5 解码

对于翻译任务，解码可以理解为翻译的过程。可以手动创建两个文件来进行测试 (当然也可以用 t2t-datagen 产生的数据)。

https://github.com/tensorflow/tensor2tensor/tree/v1.0.12

# Decode

DECODE_FILE=$DATA_DIR/decode_this.txt
echo "Hello world" >> $DECODE_FILE
echo "Goodbye world" >> $DECODE_FILE

https://github.com/tensorflow/tensor2tensor

# Decode

DECODE_FILE=$DATA_DIR/decode_this.txt
echo "Hello world" >> $DECODE_FILE
echo "Goodbye world" >> $DECODE_FILE
echo -e 'Hallo Welt\nAuf Wiedersehen Welt' > ref-translation.de

对创建好的文件进行解码 (翻译)。

https://github.com/tensorflow/tensor2tensor/tree/v1.0.12

BEAM_SIZE=4
ALPHA=0.6

t2t-trainer --data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR --train_steps=0 --eval_steps=0 --decode_beam_size=$BEAM_SIZE --decode_alpha=$ALPHA --decode_from_file=$DECODE_FILE

cat $DECODE_FILE.$MODEL.$HPARAMS.beam$BEAM_SIZE.alpha$ALPHA.decodes

https://github.com/tensorflow/tensor2tensor

BEAM_SIZE=4
ALPHA=0.6

t2t-decoder \
  --data_dir=$DATA_DIR \
  --problem=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$TRAIN_DIR \
  --decode_hparams="beam_size=$BEAM_SIZE,alpha=$ALPHA" \
  --decode_from_file=$DECODE_FILE \
  --decode_to_file=translation.en

# See the translations
cat translation.en

注意 --decode_from_file= 和 --decode_to_file 的对应文件。

3.6 评分

https://github.com/tensorflow/tensor2tensor

# Evaluate the BLEU score
# Report this BLEU score in papers, not the internal approx_bleu metric.
t2t-bleu --translation=translation.en --reference=ref-translation.de

4. Installation

# Assumes tensorflow or tensorflow-gpu installed
pip install tensor2tensor

# Installs with tensorflow-gpu requirement
pip install tensor2tensor[tensorflow_gpu]

# Installs with tensorflow (cpu) requirement
pip install tensor2tensor[tensorflow]

Binaries:

# Data generator
t2t-datagen

# Trainer
t2t-trainer --registry_help

Library usage:

python -c "from tensor2tensor.models.transformer import Transformer"

5. T2T overview

5.1 Datasets

Datasets are all standardized on TFRecord files with tensorflow.Example protocol buffers. All datasets are registered and generated with the data generator and many common sequence datasets are already available for generation and use.

https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/bin/t2t-datagen

数据集都在带有 tensorflow.Example 协议缓冲区的 TFRecord 文件上标准化。所有数据集都已通过数据生成器注册并生成，并且许多通用序列数据集已经可供生成和使用。

5.2 Problems and Modalities

Problems define training-time hyperparameters for the dataset and task, mainly by setting input and output modalities (e.g. symbol, image, audio, label) and vocabularies, if applicable. All problems are defined in problem_hparams.py. Modalities, defined in modality.py, abstract away the input and output data types so that models may deal with modality-independent tensors.

https://github.com/tensorflow/tensor2tensor/blob/v1.0.12/tensor2tensor/data_generators/problem_hparams.py
https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/utils/modality.py

Problems 定义了数据集和任务的训练时超参数，主要是通过设置输入和输出 modalities (例如符号、图像、音频、标签) 和词汇表 (如果适用) 来定义的。所有问题均在 problem_hparams.py 中定义。在 modality.py 中定义的 Modalities 可以抽象出输入和输出数据类型，以便 models 可以处理与模态无关的张量。

5.3 Models

T2TModels define the core tensor-to-tensor transformation, independent of input/output modality or task. Models take dense tensors in and produce dense tensors that may then be transformed in a final step by a modality depending on the task (e.g. fed through a final linear transform to produce logits for a softmax over classes). All models are imported in models.py, inherit from T2TModel - defined in t2t_model.py

T2TModels 定义了核心张量到张量的转换，而与输入/输出模态或任务无关。模型采用密集的张量输入并产生密集的张量，然后可以根据任务通过模态在最后一步对其进行转换 (例如，通过最终的线性变换进行馈送以产生针对类的 softmax 的对数)。所有模型均导入到 models.py 中，并继承自 T2TModel (在 t2t_model.py 中定义)。

https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/models/models.py
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/registry.py

5.4 Hyperparameter Sets

Hyperparameter sets are defined and registered in code with @registry.register_hparams and are encoded in tf.contrib.training.HParams objects. The HParams are available to both the problem specification and the model. A basic set of hyperparameters are defined in common_hparams.py and hyperparameter set functions can compose other hyperparameter set functions.

https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/utils/registry.py
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/training/python/training/hparam.py
https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/models/common_hparams.py

Hyperparameter sets 是通过 @registry.register_hparams 在代码中定义和注册的，并在tf.contrib.training.HParams 对象中进行了编码。HParams 可用于问题说明和模型。在 common_hparams.py 中定义了一组基本的超参数，并且超参数集函数可以组成其他超参数集函数。

5.5 Trainer

The trainer binary is the main entrypoint for training, evaluation, and inference. Users can easily switch between problems, models, and hyperparameter sets by using the --model, --problems, and --hparams_set flags. Specific
hyperparameters can be overridden with the --hparams flag. --schedule and related flags control local and distributed training/evaluation (distributed training documentation).

https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/docs/distributed_training.md

trainer 二进制文件是进行训练、评估和推断的主要切入点。用户可以使用 --model, --problems, and --hparams_set 标志轻松地在问题、模型和超参数集之间切换。特定的超参数可以使用 --hparams 标志覆盖。--schedule 和相关标志控制本地和分布式训练/评估 (distributed training documentation)。

References

https://www.zhihu.com/people/jin-tian-zuo-zuo-ye-mei/posts

https://blog.csdn.net/csa121/article/details/79605215
https://blog.csdn.net/qq_30650047/article/details/102996021
https://androidkt.com/training-neural-machine-translation-with-tensor2tensor/
https://blog.csdn.net/hpulfc/article/details/81172498

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Yongqiang Cheng

梦想不是浮躁，而是沉淀和积累。

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值