kaldi aishell1语音识别

最新推荐文章于 2024-09-06 23:14:41 发布
原创
最新推荐文章于 2024-09-06 23:14:41 发布 · 2.4k 阅读
6 ·
CC 4.0 BY-SA版权
文章标签：
#自然语言处理 #语音识别
本文介绍了使用Kaldi进行语音识别的详细步骤，包括下载Kaldi库并安装依赖包、编译Kaldi源码、下载及配置Aishell1数据集、修改配置文件等，并展示了训练过程中的部分日志。
最近就会一直哭，坐地铁也会哭，中午吃饭也会哭。我真的好难过。
下面来讲一讲kaldi的语音识别，使用aishell1的15GB的语音数据
1、首先要下载kaldi的语音识别库
git clone https://github.com/kaldi-asr/kaldi.git
然后进入到tools文件夹下面安装依赖包
cd tools
.extras/check_dependencies.sh
check_dependencies.sh之后会提示相关的命令，让你去安装相关的包。
安装完后再执行
.extras/check_dependencies.sh
命令
重复上述步骤，直到.extras/check_dependencies.sh显示OK为止
2、进入到kaldi/src目录下面，去编译kaldi
./configure --shared
make depend -j 8
make -j 8
3、下载aishell1的数据集去进行训练
http://www.openslr.org/resources/33/data_aishell.tgz
4、下载完后，就准备开始训练了
一、首先修改下面的这个文件
kaldi/egs/aishell/s5/cmd.sh
把下面的queue.pl 改成run.pl 。
因为是在本地跑的，所以没有集群
export train_cmd="queue.pl --mem 2G"
export decode_cmd="queue.pl --mem 4G"
export mkgraph_cmd="queue.pl --mem 8G"
二、修改下面这个文件中的数据目录的地址
kaldi/egs/aishell/s5/cmd.sh
data=/export/a05/xna/data 改成你所下载的data_aishell.tgz文件所在的文件夹就可以了
data_url=www.openslr.org/resources/33 表示还要下载一个resource_data.tgz文件，会去自动下载，这个可以不需要care它
三、最后执行run.sh命令
下面是识别的日志
nohup: ignoring input
local/nnet3/run_ivector_common.sh: training the iVector extractor
steps/online/nnet2/train_ivector_extractor.sh --cmd run.pl --mem 2G --nj 10 data/train_sp_hires_nopitch exp/nnet3/diag_ubm exp/nnet3/extractor
utils/data/modify_speaker_info.sh: copied data from data/train_sp_hires_nopitch to exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2, number of speakers changed from 1020 to 180399
utils/validate_data_dir.sh: Successfully validated data-directory exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 2G --nj 30 exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2 exp/nnet3/extractor exp/nnet3/ivectors_train_sp
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_train_sp using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 2G --nj 8 data/dev_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_dev
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_dev using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 2G --nj 8 data/test_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_test
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_test using the extractor in exp/nnet3/extractor.
local/nnet3/run_tdnn.sh: creating neural net configs
tree-info exp/tri5a_sp_ali/tree 
steps/nnet3/xconfig_to_configs.py --xconfig-file exp/nnet3/tdnn_sp/configs/network.xconfig --config-dir exp/nnet3/tdnn_sp/configs/
nnet3-init exp/nnet3/tdnn_sp/configs//init.config exp/nnet3/tdnn_sp/configs//init.raw 
LOG (nnet3-init[5.5.814~4-3ce241]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//init.raw
nnet3-info exp/nnet3/tdnn_sp/configs//init.raw 
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw 
LOG (nnet3-init[5.5.814~4-3ce241]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw 
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw 
LOG (nnet3-init[5.5.814~4-3ce241]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw 
2020-10-16 04:12:04,365 [steps/nnet3/train_dnn.py:36 - <module> - INFO ] Starting DNN trainer (train_dnn.py)
steps/nnet3/train_dnn.py --stage=-10 --cmd=run.pl --mem 4G --feat.online-ivector-dir exp/nnet3/ivectors_train_sp --feat.cmvn-opts=--norm-means=false --norm-vars=false --trainer.num-epochs 4 --trainer.optimization.num-jobs-initial 2 --trainer.optimization.num-jobs-final 2 --trainer.optimization.initial-effective-lrate 0.0015 --trainer.optimization.final-effective-lrate 0.00015 --egs.dir  --cleanup.remove-egs true --cleanup.preserve-model-interval 500 --use-gpu wait --feat-dir=data/train_sp_hires --ali-dir exp/tri5a_sp_ali --lang data/lang --reporting.email= --dir=exp/nnet3/tdnn_sp
['steps/nnet3/train_dnn.py', '--stage=-10', '--cmd=run.pl --mem 4G', '--feat.online-ivector-dir', 'exp/nnet3/ivectors_train_sp', '--feat.cmvn-opts=--norm-means=false --norm-vars=false', '--trainer.num-epochs', '4', '--trainer.optimization.num-jobs-initial', '2', '--trainer.optimization.num-jobs-final', '2', '--trainer.optimization.initial-effective-lrate', '0.0015', '--trainer.optimization.final-effective-lrate', '0.00015', '--egs.dir', '', '--cleanup.remove-egs', 'true', '--cleanup.preserve-model-interval', '500', '--use-gpu', 'wait', '--feat-dir=data/train_sp_hires', '--ali-dir', 'exp/tri5a_sp_ali', '--lang', 'data/lang', '--reporting.email=', '--dir=exp/nnet3/tdnn_sp']
2020-10-16 04:12:04,379 [steps/nnet3/train_dnn.py:178 - train - INFO ] Arguments for the experiment
{'ali_dir': 'exp/tri5a_sp_ali',
 'backstitch_training_interval': 1,
 'backstitch_training_scale': 0.0,
 'cleanup': True,
 'cmvn_opts': '--norm-means=false --norm-vars=false',
 'combine_sum_to_one_penalty': 0.0,
 'command': 'run.pl --mem 4G',
 'compute_per_dim_accuracy': False,
 'dir': 'exp/nnet3/tdnn_sp',
 'do_final_combination': True,
 'dropout_schedule': None,
 'egs_command': None,
 'egs_dir': None,
 'egs_opts': None,
 'egs_stage': 0,
 'email': None,
 'exit_stage': None,
 'feat_dir': 'data/train_sp_hires',
 'final_effective_lrate': 0.00015,
 'frames_per_eg': 8,
 'initial_effective_lrate': 0.0015,
 'input_model': None,
 'lang': 'data/lang',
 'max_lda_jobs': 10,
 'max_models_combine': 20,
 'max_objective_evaluations': 30,
 'max_param_change': 2.0,
 'minibatch_size': '512',
 'momentum': 0.0,
 'num_epochs': 4.0,
 'num_jobs_compute_prior': 10,
 'num_jobs_final': 2,
 'num_jobs_initial': 2,
 'num_jobs_step': 1,
 'online_ivector_dir': 'exp/nnet3/ivectors_train_sp',
 'preserve_model_interval': 500,
 'presoftmax_prior_scale_power': -0.25,
 'prior_subset_size': 20000,
 'proportional_shrink': 0.0,
 'rand_prune': 4.0,
 'remove_egs': True,
 'reporting_interval': 0.1,
 'samples_per_iter': 400000,
 'shuffle_buffer_size': 5000,
 'srand': 0,
 'stage': -10,
 'train_opts': [],
 'use_gpu': 'wait'}
2020-10-16 04:12:07,825 [steps/nnet3/train_dnn.py:228 - train - INFO ] Initializing a basic network for estimating preconditioning matrix
2020-10-16 04:12:07,974 [steps/nnet3/train_dnn.py:238 - train - INFO ] Generating egs
steps/nnet3/get_egs.sh --cmd run.pl --mem 4G --cmvn-opts --norm-means=false --norm-vars=false --online-ivector-dir exp/nnet3/ivectors_train_sp --left-context 16 --right-context 12 --left-context-initial -1 --right-context-final -1 --stage 0 --samples-per-iter 400000 --frames-per-eg 8 --srand 0 data/train_sp_hires exp/tri5a_sp_ali exp/nnet3/tdnn_sp/egs
File data/train_sp_hires/utt2uniq exists, so augmenting valid_uttlist to
include all perturbed versions of the same 'real' utterances.
steps/nnet3/get_egs.sh: creating egs.  To ensure they are not deleted later you can do:  touch exp/nnet3/tdnn_sp/egs/.nodelete
steps/nnet3/get_egs.sh: feature type is raw, with 'apply-cmvn'
feat-to-dim scp:exp/nnet3/ivectors_train_sp/ivector_online.scp - 
steps/nnet3/get_egs.sh: working out number of frames of training data
steps/nnet3/get_egs.sh: working out feature dim
steps/nnet3/get_egs.sh: creating 52 archives, each with 394272 egs, with
steps/nnet3/get_egs.sh:   8 labels per example, and (left,right) context = (16,12)
steps/nnet3/get_egs.sh: copying data alignments
copy-int-vector ark:- ark,scp:exp/nnet3/tdnn_sp/egs/ali.ark,exp/nnet3/tdnn_sp/egs/ali.scp 
LOG (copy-int-vector[5.5.814~4-3ce241]:main():copy-int-vector.cc:83) Copied 360291 vectors of int32.
steps/nnet3/get_egs.sh: Getting validation and training subset examples.
steps/nnet3/get_egs.sh: ... extracting validation and training-subset alignments.
... Getting subsets of validation examples for diagnostics and combination.
steps/nnet3/get_egs.sh: Generating training examples on disk
steps/nnet3/get_egs.sh: recombining and shuffling order of archives on disk
steps/nnet3/get_egs.sh: removing temporary archives
steps/nnet3/get_egs.sh: removing temporary alignments
steps/nnet3/get_egs.sh: Finished preparing training examples
2020-10-16 04:55:25,722 [steps/nnet3/train_dnn.py:276 - train - INFO ] Computing the preconditioning matrix for input features
2020-10-16 05:04:23,973 [steps/nnet3/train_dnn.py:287 - train - INFO ] Computing initial vector for FixedScaleComponent before softmax, using priors^-0.25 and rescaling to average 1
2020-10-16 05:04:33,927 [steps/nnet3/train_dnn.py:294 - train - INFO ] Preparing the initial acoustic model.
2020-10-16 05:04:44,296 [steps/nnet3/train_dnn.py:319 - train - INFO ] Training will run for 4.0 epochs = 832 iterations
2020-10-16 05:04:44,297 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 0/831   Jobs: 2   Epoch: 0.00/4.0 (0.0% complete)   lr: 0.003000   
2020-10-16 05:05:44,597 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 1/831   Jobs: 2   Epoch: 0.00/4.0 (0.1% complete)   lr: 0.002992   
2020-10-16 05:06:41,698 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 2/831   Jobs: 2   Epoch: 0.01/4.0 (0.2% complete)   lr: 0.002983   
2020-10-16 05:07:37,982 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 3/831   Jobs: 2   Epoch: 0.01/4.0 (0.4% complete)   lr: 0.002975   
2020-10-16 05:08:33,712 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 4/831   Jobs: 2   Epoch: 0.02/4.0 (0.5% complete)   lr: 0.002967   
2020-10-16 05:09:30,842 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 5/831   Jobs: 2   Epoch: 0.02/4.0 (0.6% complete)   lr: 0.002959   
2020-10-16 05:10:46,849 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 6/831   Jobs: 2   Epoch: 0.03/4.0 (0.7% complete)   lr: 0.002951   
2020-10-16 05:11:56,632 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 7/831   Jobs: 2   Epoch: 0.03/4.0 (0.8% complete)   lr: 0.002942   
2020-10-16 05:13:05,099 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 8/831   Jobs: 2   Epoch: 0.04/4.0 (1.0% complete)   lr: 0.002934   
2020-10-16 05:14:16,828 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 9/831   Jobs: 2   Epoch: 0.04/4.0 (1.1% complete)   lr: 0.002926   
2020-10-16 05:15:28,308 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 10/831   Jobs: 2   Epoch: 0.05/4.0 (1.2% complete)   lr: 0.002918   
2020-10-16 05:16:24,801 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 11/831   Jobs: 2   Epoch: 0.05/4.0 (1.3% complete)   lr: 0.002910   
2020-10-16 05:17:21,802 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 12/831   Jobs: 2   Epoch: 0.06/4.0 (1.4% complete)   lr: 0.002902   
2020-10-16 05:18:18,906 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 13/831   Jobs: 2   Epoch: 0.06/4.0 (1.6% complete)   lr: 0.002894   
2020-10-16 05:19:15,981 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 14/831   Jobs: 2   Epoch: 0.07/4.0 (1.7% complete)   lr: 0.002886   
2020-10-16 05:20:12,785 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 15/831   Jobs: 2   Epoch: 0.07/4.0 (1.8% complete)   lr: 0.002878   
2020-10-16 05:21:09,623 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 16/831   Jobs: 2   Epoch: 0.08/4.0 (1.9% complete)   lr: 0.002870   
2020-10-16 05:22:06,385 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 17/831   Jobs: 2   Epoch: 0.08/4.0 (2.0% complete)   lr: 0.002862   
2020-10-16 05:23:02,744 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 18/831   Jobs: 2   Epoch: 0.09/4.0 (2.2% complete)   lr: 0.002854   
2020-10-16 05:23:59,714 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 19/831   Jobs: 2   Epoch: 0.09/4.0 (2.3% complete)   lr: 0.002846   
2020-10-16 05:24:56,352 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 20/831   Jobs: 2   Epoch: 0.10/4.0 (2.4% complete)   lr: 0.002838   
2020-10-16 05:25:56,027 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 21/831   Jobs: 2   Epoch: 0.10/4.0 (2.5% complete)   lr: 0.002831   
2020-10-16 05:26:52,836 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 22/831   Jobs: 2   Epoch: 0.11/4.0 (2.6% complete)   lr: 0.002823   
2020-10-16 05:27:49,771 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 23/831   Jobs: 2   Epoch: 0.11/4.0 (2.8% complete)   lr: 0.002815   
2020-10-16 05:28:46,837 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 24/831   Jobs: 2   Epoch: 0.12/4.0 (2.9% complete)   lr: 0.002807   
2020-10-16 05:29:43,671 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 25/831   Jobs: 2   Epoch: 0.12/4.0 (3.0% complete)   lr: 0.002799   
2020-10-16 05:30:40,848 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 26/831   Jobs: 2   Epoch: 0.12/4.0 (3.1% complete)   lr: 0.002792   
2020-10-16 05:31:37,973 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 27/831   Jobs: 2   Epoch: 0.13/4.0 (3.2% complete)   lr: 0.002784   
2020-10-16 05:32:35,122 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 28/831   Jobs: 2   Epoch: 0.13/4.0 (3.4% complete)   lr: 0.002776   
2020-10-16 05:33:31,688 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 29/831   Jobs: 2   Epoch: 0.14/4.0 (3.5% complete)   lr: 0.002769   
2020-10-16 05:34:28,835 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 30/831   Jobs: 2   Epoch: 0.14/4.0 (3.6% complete)   lr: 0.002761   
2020-10-16 05:35:25,719 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 31/831   Jobs: 2   Epoch: 0.15/4.0 (3.7% complete)   lr: 0.002753   
2020-10-16 05:36:23,054 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 32/831   Jobs: 2   Epoch: 0.15/4.0 (3.8% complete)   lr: 0.002746   
2020-10-16 05:37:19,915 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 33/831   Jobs: 2   Epoch: 0.16/4.0 (4.0% complete)   lr: 0.002738   
2020-10-16 05:38:16,925 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 34/831   Jobs: 2   Epoch: 0.16/4.0 (4.1% complete)   lr: 0.002731   
2020-10-16 05:39:13,181 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 35/831   Jobs: 2   Epoch: 0.17/4.0 (4.2% complete)   lr: 0.002723   
2020-10-16 05:40:10,302 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 36/831   Jobs: 2   Epoch: 0.17/4.0 (4.3% complete)   lr: 0.002716   
2020-10-16 05:41:07,285 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 37/831   Jobs: 2   Epoch: 0.18/4.0 (4.4% complete)   lr: 0.002708   
2020-10-16 05:42:04,467 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 38/831   Jobs: 2   Epoch: 0.18/4.0 (4.6% complete)   lr: 0.002701   
2020-10-16 05:43:00,830 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 39/831   Jobs: 2   Epoch: 0.19/4.0 (4.7% complete)   lr: 0.002693   
2020-10-16 05:43:57,886 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 40/831   Jobs: 2   Epoch: 0.19/4.0 (4.8% complete)   lr: 0.002686   
2020-10-16 05:44:57,538 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 41/831   Jobs: 2   Epoch: 0.20/4.0 (4.9% complete)   lr: 0.002678   
2020-10-16 05:45:54,379 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 42/831   Jobs: 2   Epoch: 0.20/4.0 (5.0% complete)   lr: 0.002671   
2020-10-16 05:46:50,807 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 43/831   Jobs: 2   Epoch: 0.21/4.0 (5.2% complete)   lr: 0.002663   
2020-10-16 05:47:47,542 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 44/831   Jobs: 2   Epoch: 0.21/4.0 (5.3% complete)   lr: 0.002656   
2020-10-16 05:48:44,027 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 45/831   Jobs: 2   Epoch: 0.22/4.0 (5.4% complete)   lr: 0.002649   
2020-10-16 05:49:41,102 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 46/831   Jobs: 2   Epoch: 0.22/4.0 (5.5% complete)   lr: 0.002641   
2020-10-16 05:50:38,021 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 47/831   Jobs: 2   Epoch: 0.23/4.0 (5.6% complete)   lr: 0.002634   
2020-10-16 05:51:34,709 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 48/831   Jobs: 2   Epoch: 0.23/4.0 (5.8% complete)   lr: 0.002627   
2020-10-16 05:52:31,832 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 49/831   Jobs: 2   Epoch: 0.24/4.0 (5.9% complete)   lr: 0.002620   
2020-10-16 05:53:27,885 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 50/831   Jobs: 2   Epoch: 0.24/4.0 (6.0% complete)   lr: 0.002612   
2020-10-16 05:54:24,876 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 51/831   Jobs: 2   Epoch: 0.25/4.0 (6.1% complete)   lr: 0.002605   
2020-10-16 05:55:20,847 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 52/831   Jobs: 2   Epoch: 0.25/4.0 (6.2% complete)   lr: 0.002598   
2020-10-16 05:56:17,213 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 53/831   Jobs: 2   Epoch: 0.25/4.0 (6.4% complete)   lr: 0.002591   
2020-10-16 05:57:12,977 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 54/831   Jobs: 2   Epoch: 0.26/4.0 (6.5% complete)   lr: 0.002584   
2020-10-16 05:58:09,984 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 55/831   Jobs: 2   Epoch: 0.26/4.0 (6.6% complete)   lr: 0.002576   
2020-10-16 05:59:06,743 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 56/831   Jobs: 2   Epoch: 0.27/4.0 (6.7% complete)   lr: 0.002569   
2020-10-16 06:00:02,946 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 57/831   Jobs: 2   Epoch: 0.27/4.0 (6.9% complete)   lr: 0.002562   
2020-10-16 06:00:59,436 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 58/831   Jobs: 2   Epoch: 0.28/4.0 (7.0% complete)   lr: 0.002555   
2020-10-16 06:01:56,221 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 59/831   Jobs: 2   Epoch: 0.28/4.0 (7.1% complete)   lr: 0.002548   
2020-10-16 06:02:52,520 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 60/831   Jobs: 2   Epoch: 0.29/4.0 (7.2% complete)   lr: 0.002541   
2020-10-16 06:03:52,055 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 61/831   Jobs: 2   Epoch: 0.29/4.0 (7.3% complete)   lr: 0.002534   
2020-10-16 06:04:47,626 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 62/831   Jobs: 2   Epoch: 0.30/4.0 (7.5% complete)   lr: 0.002527   
2020-10-16 06:05:43,905 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 63/831   Jobs: 2   Epoch: 0.30/4.0 (7.6% complete)   lr: 0.002520   
2020-10-16 06:06:40,608 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 64/831   Jobs: 2   Epoch: 0.31/4.0 (7.7% complete)   lr: 0.002513   
2020-10-16 06:07:37,478 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 65/831   Jobs: 2   Epoch: 0.31/4.0 (7.8% complete)   lr: 0.002506   
2020-10-16 06:08:34,606 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 66/831   Jobs: 2   Epoch: 0.32/4.0 (7.9% complete)   lr: 0.002499   
2020-10-16 06:09:30,998 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 67/831   Jobs: 2   Epoch: 0.32/4.0 (8.1% complete)   lr: 0.002492   
2020-10-16 06:10:26,981 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 68/831   Jobs: 2   Epoch: 0.33/4.0 (8.2% complete)   lr: 0.002485   
2020-10-16 06:11:22,411 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 69/831   Jobs: 2   Epoch: 0.33/4.0 (8.3% complete)   lr: 0.002478   
2020-10-16 06:12:19,313 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 70/831   Jobs: 2   Epoch: 0.34/4.0 (8.4% complete)   lr: 0.002472   
2020-10-16 06:13:16,613 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 71/831   Jobs: 2   Epoch: 0.34/4.0 (8.5% complete)   lr: 0.002465   
2020-10-16 06:14:13,919 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 72/831   Jobs: 2   Epoch: 0.35/4.0 (8.7% complete)   lr: 0.002458   
2020-10-16 06:15:10,097 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 73/831   Jobs: 2   Epoch: 0.35/4.0 (8.8% complete)   lr: 0.002451   
2020-10-16 06:16:06,754 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 74/831   Jobs: 2   Epoch: 0.36/4.0 (8.9% complete)   lr: 0.002444   
2020-10-16 06:17:03,935 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 75/831   Jobs: 2   Epoch: 0.36/4.0 (9.0% complete)   lr: 0.002438   
2020-10-16 06:18:01,050 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 76/831   Jobs: 2   Epoch: 0.37/4.0 (9.1% complete)   lr: 0.002431   
2020-10-16 06:18:57,946 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 77/831   Jobs: 2   Epoch: 0.37/4.0 (9.3% complete)   lr: 0.002424   
2020-10-16 06:19:54,394 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 78/831   Jobs: 2   Epoch: 0.38/4.0 (9.4% complete)   lr: 0.002418   
2020-10-16 06:20:50,941 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 79/831   Jobs: 2   Epoch: 0.38/4.0 (9.5% complete)   lr: 0.002411   
2020-10-16 06:21:47,574 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 80/831   Jobs: 2   Epoch: 0.38/4.0 (9.6% complete)   lr: 0.002404   
2020-10-16 06:22:47,573 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 81/831   Jobs: 2   Epoch: 0.39/4.0 (9.7% complete)   lr: 0.002398   
2020-10-16 06:23:43,185 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 82/831   Jobs: 2   Epoch: 0.39/4.0 (9.9% complete)   lr: 0.002391   
2020-10-16 06:24:39,704 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 83/831   Jobs: 2   Epoch: 0.40/4.0 (10.0% complete)   lr: 0.002384   
2020-10-16 06:25:35,503 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 84/831   Jobs: 2   Epoch: 0.40/4.0 (10.1% complete)   lr: 0.002378   
2020-10-16 06:26:31,770 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 85/831   Jobs: 2   Epoch: 0.41/4.0 (10.2% complete)   lr: 0.002371   
2020-10-16 06:27:28,527 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 86/831   Jobs: 2   Epoch: 0.41/4.0 (10.3% complete)   lr: 0.002365   
2020-10-16 06:28:25,726 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 87/831   Jobs: 2   Epoch: 0.42/4.0 (10.5% complete)   lr: 0.002358   
2020-10-16 06:29:22,161 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 88/831   Jobs: 2   Epoch: 0.42/4.0 (10.6% complete)   lr: 0.002352   
2020-10-16 06:30:18,811 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 89/831   Jobs: 2   Epoch: 0.43/4.0 (10.7% complete)   lr: 0.002345   
2020-10-16 06:31:15,519 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 90/831   Jobs: 2   Epoch: 0.43/4.0 (10.8% complete)   lr: 0.002339   
2020-10-16 06:32:12,616 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 91/831   Jobs: 2   Epoch: 0.44/4.0 (10.9% complete)   lr: 0.002332   
2020-10-16 06:33:09,387 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 92/831   Jobs: 2   Epoch: 0.44/4.0 (11.1% complete)   lr: 0.002326   
2020-10-16 06:34:06,491 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 93/831   Jobs: 2   Epoch: 0.45/4.0 (11.2% complete)   lr: 0.002319   
2020-10-16 06:35:03,553 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 94/831   Jobs: 2   Epoch: 0.45/4.0 (11.3% complete)   lr: 0.002313   
2020-10-16 06:35:59,697 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 95/831   Jobs: 2   Epoch: 0.46/4.0 (11.4% complete)   lr: 0.002306   
2020-10-16 06:36:56,576 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 96/831   Jobs: 2   Epoch: 0.46/4.0 (11.5% complete)   lr: 0.002300   
2020-10-16 06:37:52,431 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 97/831   Jobs: 2   Epoch: 0.47/4.0 (11.7% complete)   lr: 0.002294   
2020-10-16 06:38:49,565 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 98/831   Jobs: 2   Epoch: 0.47/4.0 (11.8% complete)   lr: 0.002287   
2020-10-16 06:39:45,406 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 99/831   Jobs: 2   Epoch: 0.48/4.0 (11.9% complete)   lr: 0.002281   
2020-10-16 06:40:42,488 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 100/831   Jobs: 2   Epoch: 0.48/4.0 (12.0% complete)   lr: 0.002275   
2020-10-16 06:41:41,379 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 101/831   Jobs: 2   Epoch: 0.49/4.0 (12.1% complete)   lr: 0.002268   
2020-10-16 06:42:38,194 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 102/831   Jobs: 2   Epoch: 0.49/4.0 (12.3% complete)   lr: 0.002262   
2020-10-16 06:43:35,145 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 103/831   Jobs: 2   Epoch: 0.50/4.0 (12.4% complete)   lr: 0.002256   
2020-10-16 06:44:31,501 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 104/831   Jobs: 2   Epoch: 0.50/4.0 (12.5% complete)   lr: 0.002250   
2020-10-16 06:45:27,371 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 105/831   Jobs: 2   Epoch: 0.50/4.0 (12.6% complete)   lr: 0.002243   
2020-10-16 06:46:24,576 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 106/831   Jobs: 2   Epoch: 0.51/4.0 (12.7% complete)   lr: 0.002237   
2020-10-16 06:47:20,883 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 107/831   Jobs: 2   Epoch: 0.51/4.0 (12.9% complete)   lr: 0.002231   
2020-10-16 06:48:17,068 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 108/831   Jobs: 2   Epoch: 0.52/4.0 (13.0% complete)   lr: 0.002225   
2020-10-16 06:49:13,871 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 109/831   Jobs: 2   Epoch: 0.52/4.0 (13.1% complete)   lr: 0.002219   
2020-10-16 06:50:10,426 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 110/831   Jobs: 2   Epoch: 0.53/4.0 (13.2% complete)   lr: 0.002213   
2020-10-16 06:51:07,305 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 111/831   Jobs: 2   Epoch: 0.53/4.0 (13.3% complete)   lr: 0.002207   
2020-10-16 06:52:04,208 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 112/831   Jobs: 2   Epoch: 0.54/4.0 (13.5% complete)   lr: 0.002200   
2020-10-16 06:53:00,857 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 113/831   Jobs: 2   Epoch: 0.54/4.0 (13.6% complete)   lr: 0.002194   
2020-10-16 06:53:58,479 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 114/831   Jobs: 2   Epoch: 0.55/4.0 (13.7% complete)   lr: 0.002188   
2020-10-16 06:54:55,251 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 115/831   Jobs: 2   Epoch: 0.55/4.0 (13.8% complete)   lr: 0.002182   
2020-10-16 06:55:52,139 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 116/831   Jobs: 2   Epoch: 0.56/4.0 (13.9% complete)   lr: 0.002176   
2020-10-16 06:56:48,905 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 117/831   Jobs: 2   Epoch: 0.56/4.0 (14.1% complete)   lr: 0.002170   
2020-10-16 06:57:45,985 [steps/nnet3/train_dnn.py:355 - train - INFO ] Iter: 118/831   Jobs: 2   Epoch: 0.57/4.0 (14.2% complete)   lr: 0.002164   
2020-10-16 06:58:42,766 [steps/nnet3/train_dnn.py:355 - train -