kaldi做aishell，其中倒数第3步是local/nnet3/run_tdnn.sh，一共花了44个小时，
主机配置32G内存，E3 1231 V3，2个gtx950 2G的显卡，机械硬盘。
其中只有steps/nnet3/train_dnn.py这个步骤需要用到GPU，也是最耗时的步骤，花了38个小时。对显存容量要求不高。
注意一定要把local/nnet3/run_tdnn.sh中第96行的--use-gpu true修改为--use-gpu wait，因为在nvidia显卡EXCLUSIVE_PROCESS模式下，最后需要12块显卡才能支持，因为最后并发数量是12。当修改为--use-gpu wait，那么并发作业拿不到显卡，程序会等待，而不是直接报错。
local/nnet3/run_tdnn.sh完整的运行日志如下
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train/utt2dur already exists with the expected length. We won't recompute it.
utils/data/get_reco2dur.sh: data/train/reco2dur already exists with the expected length. We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train, in data/train_sp_speed0.9
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_speed0.9
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train, in data/train_sp_speed1.1
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_speed1.1
utils/data/combine_data.sh data/train_sp data/train data/train_sp_speed0.9 data/train_sp_speed1.1
utils/data/combine_data.sh: combined utt2uniq
utils/data/combine_data.sh [info]: not combining segments as it does not exist
utils/data/combine_data.sh: combined utt2spk
utils/data/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/data/combine_data.sh: combined utt2dur
utils/data/combine_data.sh: combined reco2dur
utils/data/combine_data.sh [info]: **not combining feats.scp as it does not exist everywhere**
utils/data/combine_data.sh: combined text
utils/data/combine_data.sh [info]: **not combining cmvn.scp as it does not exist everywhere**
utils/data/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/data/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/data/combine_data.sh: combined wav.scp
utils/data/combine_data.sh [info]: not combining spk2gender as it does not exist
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: generated 3-way speed-perturbed version of data in data/train, in data/train_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp
local/nnet3/run_ivector_common.sh: making MFCC features for low-resolution speed-perturbed data
steps/make_mfcc_pitch.sh --cmd run.pl --mem 24G --nj 70 data/train_sp exp/make_mfcc/train_sp mfcc_perturbed
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for train_sp
steps/compute_cmvn_stats.sh data/train_sp exp/make_mfcc/train_sp mfcc_perturbed
Succeeded creating CMVN stats for train_sp
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp/.backup
local/nnet3/run_ivector_common.sh: aligning with the perturbed low-resolution data
steps/align_fmllr.sh --nj 30 --cmd run.pl --mem 24G data/train_sp data/lang exp/tri5a exp/tri5a_sp_ali
steps/align_fmllr.sh: feature type is lda
steps/align_fmllr.sh: compiling training graphs
steps/align_fmllr.sh: aligning data in data/train_sp using exp/tri5a/final.alimdl and speaker-independent features.
steps/align_fmllr.sh: computing fMLLR transforms
steps/align_fmllr.sh: doing final alignment.
steps/align_fmllr.sh: done aligning data.
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 24G data/lang exp/tri5a_sp_ali
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri5a_sp_ali/log/analyze_alignments.log
413 warnings in exp/tri5a_sp_ali/log/align_pass1.*.log
382 warnings in exp/tri5a_sp_ali/log/align_pass2.*.log
2 warnings in exp/tri5a_sp_ali/log/fmllr.*.log
local/nnet3/run_ivector_common.sh: creating high-resolution MFCC features
utils/copy_data_dir.sh: copied data from data/train_sp to data/train_sp_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires
utils/copy_data_dir.sh: copied data from data/dev to data/dev_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires
utils/copy_data_dir.sh: copied data from data/test to data/test_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires
utils/data/perturb_data_dir_volume.sh: data/train_sp_hires/feats.scp exists; moving it to data/train_sp_hires/.backup/ as it wouldn't be valid any more.
utils/data/perturb_data_dir_volume.sh: added volume perturbation to the data in data/train_sp_hires
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/train_sp_hires exp/make_hires/train_sp mfcc_perturbed_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for train_sp_hires
steps/compute_cmvn_stats.sh data/train_sp_hires exp/make_hires/train_sp mfcc_perturbed_hires
Succeeded creating CMVN stats for train_sp_hires
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp_hires/.backup
utils/copy_data_dir.sh: copied data from data/train_sp_hires to data/train_sp_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/train_sp_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires_nopitch
steps/compute_cmvn_stats.sh data/train_sp_hires_nopitch exp/make_hires/train_sp mfcc_perturbed_hires
Succeeded creating CMVN stats for train_sp_hires_nopitch
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/dev_hires exp/make_hires/dev mfcc_perturbed_hires
steps/make_mfcc_pitch.sh: moving data/dev_hires/feats.scp to data/dev_hires/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for dev_hires
steps/compute_cmvn_stats.sh data/dev_hires exp/make_hires/dev mfcc_perturbed_hires
Succeeded creating CMVN stats for dev_hires
fix_data_dir.sh: kept all 14326 utterances.
fix_data_dir.sh: old files are kept in data/dev_hires/.backup
utils/copy_data_dir.sh: copied data from data/dev_hires to data/dev_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/dev_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires_nopitch
steps/compute_cmvn_stats.sh data/dev_hires_nopitch exp/make_hires/dev mfcc_perturbed_hires
Succeeded creating CMVN stats for dev_hires_nopitch
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/test_hires exp/make_hires/test mfcc_perturbed_hires
steps/make_mfcc_pitch.sh: moving data/test_hires/feats.scp to data/test_hires/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for test_hires
steps/compute_cmvn_stats.sh data/test_hires exp/make_hires/test mfcc_perturbed_hires
Succeeded creating CMVN stats for test_hires
fix_data_dir.sh: kept all 7176 utterances.
fix_data_dir.sh: old files are kept in data/test_hires/.backup
utils/copy_data_dir.sh: copied data from data/test_hires to data/test_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/test_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires_nopitch
steps/compute_cmvn_stats.sh data/test_hires_nopitch exp/make_hires/test mfcc_perturbed_hires
Succeeded creating CMVN stats for test_hires_nopitch
local/nnet3/run_ivector_common.sh: computing a subset of data to train the diagonal UBM.
utils/data/subset_data_dir.sh: reducing #utt from 360294 to 90073
local/nnet3/run_ivector_common.sh: computing a PCA transform from the hires data.
steps/online/nnet2/get_pca_transform.sh --cmd run.pl --mem 24G --splice-opts --left-context=3 --right-context=3 --max-utts 10000 --subsample 2 exp/nnet3/diag_ubm/train_sp_hires_nopitch_subset exp/nnet3/pca_transform
Done estimating PCA transform in exp/nnet3/pca_transform
local/nnet3/run_ivector_common.sh: training the diagonal UBM.
steps/online/nnet2/train_diag_ubm.sh --cmd run.pl --mem 24G --nj 30 --num-frames 700000 --num-threads 8 exp/nnet3/diag_ubm/train_sp_hires_nopitch_subset 512 exp/nnet3/pca_transform exp/nnet3/diag_ubm
steps/online/nnet2/train_diag_ubm.sh: Directory exp/nnet3/diag_ubm already exists. Backing up diagonal UBM in exp/nnet3/diag_ubm/backup.sth
steps/online/nnet2/train_diag_ubm.sh: initializing model from E-M in memory,
steps/online/nnet2/train_diag_ubm.sh: starting from 256 Gaussians, reaching 512;
steps/online/nnet2/train_diag_ubm.sh: for 20 iterations, using at most 700000 frames of data
Getting Gaussian-selection info
steps/online/nnet2/train_diag_ubm.sh: will train for 4 iterations, in parallel over
steps/online/nnet2/train_diag_ubm.sh: 30 machines, parallelized with 'run.pl --mem 24G'
steps/online/nnet2/train_diag_ubm.sh: Training pass 0
steps/online/nnet2/train_diag_ubm.sh: Training pass 1
steps/online/nnet2/train_diag_ubm.sh: Training pass 2
steps/online/nnet2/train_diag_ubm.sh: Training pass 3
local/nnet3/run_ivector_common.sh: training the iVector extractor
steps/online/nnet2/train_ivector_extractor.sh --cmd run.pl --mem 24G --nj 10 data/train_sp_hires_nopitch exp/nnet3/diag_ubm exp/nnet3/extractor
steps/online/nnet2/train_ivector_extractor.sh: Directory exp/nnet3/extractor already exists. Backing up iVector extractor in exp/nnet3/extractor/backup.Raq
steps/online/nnet2/train_ivector_extractor.sh: doing Gaussian selection and posterior computation
Accumulating stats (pass 0)
Summing accs (pass 0)
Updating model (pass 0)
Accumulating stats (pass 1)
Summing accs (pass 1)
Updating model (pass 1)
Accumulating stats (pass 2)
Summing accs (pass 2)
Updating model (pass 2)
Accumulating stats (pass 3)
Summing accs (pass 3)
Updating model (pass 3)
Accumulating stats (pass 4)
Summing accs (pass 4)
Updating model (pass 4)
Accumulating stats (pass 5)
Summing accs (pass 5)
Updating model (pass 5)
Accumulating stats (pass 6)
Summing accs (pass 6)
Updating model (pass 6)
Accumulating stats (pass 7)
Summing accs (pass 7)
Updating model (pass 7)
Accumulating stats (pass 8)
Summing accs (pass 8)
Updating model (pass 8)
Accumulating stats (pass 9)
Summing accs (pass 9)
Updating model (pass 9)
utils/data/modify_speaker_info.sh: copied data from data/train_sp_hires_nopitch to exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2, number of speakers changed from 1020 to 180399
utils/validate_data_dir.sh: Successfully validated data-directory exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 30 exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2 exp/nnet3/extractor exp/nnet3/ivectors_train_sp
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_train_sp using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 8 data/dev_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_dev
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_dev using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 8 data/test_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_test
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_test using the extractor in exp/nnet3/extractor.
local/nnet3/run_tdnn.sh: creating neural net configs
tree-info exp/tri5a_sp_ali/tree
steps/nnet3/xconfig_to_configs.py --xconfig-file exp/nnet3/tdnn_sp/configs/network.xconfig --config-dir exp/nnet3/tdnn_sp/configs/
nnet3-init exp/nnet3/tdnn_sp/configs//init.config exp/nnet3/tdnn_sp/configs//init.raw
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//init.raw
nnet3-info exp/nnet3/tdnn_sp/configs//init.raw
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
2019-04-05 22:08:04,965 [steps/nnet3/train_dnn.py:36 - <module> - INFO ] Starting DNN trainer (train_dnn.py)
steps/nnet3/train_dnn.py --stage=-10 --cmd=run.pl --mem 24G --feat.online-ivector-dir exp/nnet3/ivectors_train_sp --feat.cmvn-opts=--norm-means=false --norm-vars=false --trainer.num-epochs 4 --trainer.optimization.num-jobs-initial 2 --trainer.optimization.num-jobs-final 12 --trainer.optimization.initial-effective-lrate 0.0015 --trainer.optimization.final-effective-lrate 0.00015 --egs.dir --cleanup.remove-egs true --cleanup.preserve-model-interval 500 --use-gpu wait --feat-dir=data/train_sp_hires --ali-dir exp/tri5a_sp_ali --lang data/lang --reporting.email= --dir=exp/nnet3/tdnn_sp
['steps/nnet3/train_dnn.py', '--stage=-10', '--cmd=run.pl --mem 24G', '--feat.online-ivector-dir', 'exp/nnet3/ivectors_train_sp', '--feat.cmvn-opts=--norm-means=false --norm-vars=false', '--trainer.num-epochs', '4', '--trainer.optimization.num-jobs-initial', '2', '--trainer.optimization.num-jobs-final', '12', '--trainer.optimization.initial-effective-lrate', '0.0015', '--trainer.optimization.final-effective-lrate', '0.00015', '--egs.dir', '', '--cleanup.remove-egs', 'true', '--cleanup.preserve-model-interval', '500', '--use-gpu', 'wait', '--feat-dir=data/train_sp_hires', '--ali-dir', 'exp/tri5a_sp_ali', '--lang', 'data/lang', '--reporting.email=', '--dir=exp/nnet3/tdnn_sp']
2019-04-05 22:08:04,970 [steps/nnet3/train_dnn.py:177 - train - INFO ] Arguments for the experiment
{'ali_dir': 'exp/tri5a_sp_ali',
'backstitch_training_interval': 1,
'backstitch_training_scale': 0.0,
'cleanup': True,
'cmvn_opts': '--norm-means=false --norm-vars=false',
'combine_sum_to_one_penalty': 0.0,
'command': 'run.pl --mem 24G',
'compute_per_dim_accuracy': False,
'dir': 'exp/nnet3/tdnn_sp',
'do_final_combination': True,
'dropout_schedule': None,
'egs_command': None,
'egs_dir': None,
'egs_opts': None,
'egs_stage': 0,
'email': None,
'exit_stage': None,
'feat_dir': 'data/train_sp_hires',
'final_effective_lrate': 0.00015,
'frames_per_eg': 8,
'initial_effective_lrate': 0.0015,
'input_model': None,
'lang': 'data/lang',
'max_lda_jobs': 10,
'max_models_combine': 20,
'max_objective_evaluations': 30,
'max_param_change': 2.0,
'minibatch_size': '512',
'momentum': 0.0,
'num_epochs': 4.0,
'num_jobs_compute_prior': 10,
'num_jobs_final': 12,
'num_jobs_initial': 2,
'online_ivector_dir': 'exp/nnet3/ivectors_train_sp',
'preserve_model_interval': 500,
'presoftmax_prior_scale_power': -0.25,
'prior_subset_size': 20000,
'proportional_shrink': 0.0,
'rand_prune': 4.0,
'remove_egs': True,
'reporting_interval': 0.1,
'samples_per_iter': 400000,
'shuffle_buffer_size': 5000,
'srand': 0,
'stage': -10,
'train_opts': [],
'use_gpu': 'wait'}
2019-04-05 22:08:05,016 [steps/nnet3/train_dnn.py:227 - train - INFO ] Initializing a basic network for estimating preconditioning matrix
2019-04-05 22:08:05,063 [steps/nnet3/train_dnn.py:237 - train - INFO ] Generating egs
steps/nnet3/get_egs.sh --cmd run.pl --mem 24G --cmvn-opts --norm-means=false --norm-vars=false --online-ivector-dir exp/nnet3/ivectors_train_sp --left-context 16 --right-context 12 --left-context-initial -1 --right-context-final -1 --stage 0 --samples-per-iter 400000 --frames-per-eg 8 --srand 0 data/train_sp_hires exp/tri5a_sp_ali exp/nnet3/tdnn_sp/egs
File data/train_sp_hires/utt2uniq exists, so augmenting valid_uttlist to
include all perturbed versions of the same 'real' utterances.
steps/nnet3/get_egs.sh: creating egs. To ensure they are not deleted later you can do: touch exp/nnet3/tdnn_sp/egs/.nodelete
steps/nnet3/get_egs.sh: feature type is raw
feat-to-dim scp:exp/nnet3/ivectors_train_sp/ivector_online.scp -
steps/nnet3/get_egs.sh: working out number of frames of training data
steps/nnet3/get_egs.sh: working out feature dim
steps/nnet3/get_egs.sh: creating 52 archives, each with 394272 egs, with
steps/nnet3/get_egs.sh: 8 labels per example, and (left,right) context = (16,12)
steps/nnet3/get_egs.sh: copying data alignments
copy-int-vector ark:- ark,scp:exp/nnet3/tdnn_sp/egs/ali.ark,exp/nnet3/tdnn_sp/egs/ali.scp
LOG (copy-int-vector[5.5]:main():copy-int-vector.cc:83) Copied 360291 vectors of int32.
steps/nnet3/get_egs.sh: Getting validation and training subset examples.
steps/nnet3/get_egs.sh: ... extracting validation and training-subset alignments.
... Getting subsets of validation examples for diagnostics and combination.
steps/nnet3/get_egs.sh: Generating training examples on disk
steps/nnet3/get_egs.sh: recombining and shuffling order of archives on disk
steps/nnet3/get_egs.sh: removing temporary archives
steps/nnet3/get_egs.sh: removing temporary alignments
steps/nnet3/get_egs.sh: Finished preparing training examples
2019-04-05 22:49:03,244 [steps/nnet3/train_dnn.py:275 - train - INFO ] Computing the preconditioning matrix for input features
2019-04-05 22:53:51,530 [steps/nnet3/train_dnn.py:286 - train - INFO ] Computing initial vector for FixedScaleComponent before softmax, using priors^-0.25 and rescaling to average 1
2019-04-05 22:54:12,361 [steps/nnet3/train_dnn.py:293 - train - INFO ] Preparing the initial acoustic model.
2019-04-05 22:54:19,160 [steps/nnet3/train_dnn.py:318 - train - INFO ] Training will run for 4.0 epochs = 237 iterations
2019-04-05 22:54:19,160 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 0/236 Epoch: 0.00/4.0 (0.0% complete) lr: 0.003000
2019-04-05 22:57:07,706 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 1/236 Epoch: 0.00/4.0 (0.1% complete) lr: 0.002992
2019-04-05 22:59:52,272 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 2/236 Epoch: 0.01/4.0 (0.2% complete) lr: 0.002983
2019-04-05 23:02:37,357 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 3/236 Epoch: 0.01/4.0 (0.4% complete) lr: 0.002975
2019-04-05 23:05:25,685 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 4/236 Epoch: 0.02/4.0 (0.5% complete) lr: 0.002967
2019-04-05 23:08:09,379 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 5/236 Epoch: 0.02/4.0 (0.6% complete) lr: 0.002959
2019-04-05 23:10:53,866 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 6/236 Epoch: 0.03/4.0 (0.7% complete) lr: 0.002951
2019-04-05 23:13:38,331 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 7/236 Epoch: 0.03/4.0 (0.8% complete) lr: 0.002942
2019-04-05 23:16:23,374 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 8/236 Epoch: 0.04/4.0 (1.0% complete) lr: 0.002934
2019-04-05 23:19:10,803 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 9/236 Epoch: 0.04/4.0 (1.1% complete) lr: 0.002926
2019-04-05 23:21:56,003 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 10/236 Epoch: 0.05/4.0 (1.2% complete) lr: 0.002918
2019-04-05 23:24:42,471 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 11/236 Epoch: 0.05/4.0 (1.3% complete) lr: 0.002910
2019-04-05 23:27:29,937 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 12/236 Epoch: 0.06/4.0 (1.4% complete) lr: 0.004353
2019-04-05 23:32:39,287 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 13/236 Epoch: 0.06/4.0 (1.6% complete) lr: 0.004335
2019-04-05 23:37:48,546 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 14/236 Epoch: 0.07/4.0 (1.8% complete) lr: 0.004317
2019-04-05 23:42:57,747 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 15/236 Epoch: 0.08/4.0 (2.0% complete) lr: 0.004299
2019-04-05 23:48:06,678 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 16/236 Epoch: 0.09/4.0 (2.2% complete) lr: 0.004281
2019-04-05 23:53:21,294 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 17/236 Epoch: 0.09/4.0 (2.3% complete) lr: 0.004264
2019-04-05 23:58:35,909 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 18/236 Epoch: 0.10/4.0 (2.5% complete) lr: 0.004246
2019-04-06 00:03:50,992 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 19/236 Epoch: 0.11/4.0 (2.7% complete) lr: 0.004228
2019-04-06 00:09:06,003 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 20/236 Epoch: 0.12/4.0 (2.9% complete) lr: 0.004211
2019-04-06 00:14:23,242 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 21/236 Epoch: 0.12/4.0 (3.1% complete) lr: 0.004193
2019-04-06 00:19:38,874 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 22/236 Epoch: 0.13/4.0 (3.2% complete) lr: 0.004176
2019-04-06 00:24:54,063 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 23/236 Epoch: 0.14/4.0 (3.4% complete) lr: 0.004159
2019-04-06 00:30:03,795 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 24/236 Epoch: 0.14/4.0 (3.6% complete) lr: 0.004141
2019-04-06 00:35:14,407 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 25/236 Epoch: 0.15/4.0 (3.8% complete) lr: 0.004124
2019-04-06 00:40:24,596 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 26/236 Epoch: 0.16/4.0 (4.0% complete) lr: 0.004107
2019-04-06 00:45:35,149 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 27/236 Epoch: 0.17/4.0 (4.1% complete) lr: 0.004090
2019-04-06 00:50:50,207 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 28/236 Epoch: 0.17/4.0 (4.3% complete) lr: 0.004073
2019-04-06 00:56:05,661 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 29/236 Epoch: 0.18/4.0 (4.5% complete) lr: 0.004056
2019-04-06 01:01:20,715 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 30/236 Epoch: 0.19/4.0 (4.7% complete) lr: 0.004040
2019-04-06 01:06:36,168 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 31/236 Epoch: 0.19/4.0 (4.9% complete) lr: 0.004023
2019-04-06 01:11:46,296 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 32/236 Epoch: 0.20/4.0 (5.0% complete) lr: 0.004006
2019-04-06 01:16:56,093 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 33/236 Epoch: 0.21/4.0 (5.2% complete) lr: 0.003990
2019-04-06 01:22:11,559 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 34/236 Epoch: 0.22/4.0 (5.4% complete) lr: 0.003973
2019-04-06 01:27:21,740 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 35/236 Epoch: 0.22/4.0 (5.6% complete) lr: 0.003957
2019-04-06 01:32:32,328 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 36/236 Epoch: 0.23/4.0 (5.8% complete) lr: 0.005254
2019-04-06 01:37:57,318 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 37/236 Epoch: 0.24/4.0 (6.0% complete) lr: 0.005225
2019-04-06 01:43:26,665 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 38/236 Epoch: 0.25/4.0 (6.2% complete) lr: 0.005196
2019-04-06 01:48:55,859 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 39/236 Epoch: 0.26/4.0 (6.5% complete) lr: 0.005167
2019-04-06 01:54:24,345 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 40/236 Epoch: 0.27/4.0 (6.7% complete) lr: 0.005139
2019-04-06 02:00:00,250 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 41/236 Epoch: 0.28/4.0 (7.0% complete) lr: 0.005110
2019-04-06 02:05:28,891 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 42/236 Epoch: 0.29/4.0 (7.2% complete) lr: 0.005082
2019-04-06 02:10:57,812 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 43/236 Epoch: 0.30/4.0 (7.5% complete) lr: 0.005054
2019-04-06 02:16:26,757 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 44/236 Epoch: 0.31/4.0 (7.7% complete) lr: 0.005026
2019-04-06 02:21:56,025 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 45/236 Epoch: 0.32/4.0 (7.9% complete) lr: 0.004998
2019-04-06 02:27:24,615 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 46/236 Epoch: 0.33/4.0 (8.2% complete) lr: 0.004971
2019-04-06 02:32:52,893 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 47/236 Epoch: 0.34/4.0 (8.4% complete) lr: 0.004943
2019-04-06 02:38:21,770 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 48/236 Epoch: 0.35/4.0 (8.7% complete) lr: 0.004916
2019-04-06 02:43:50,027 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 49/236 Epoch: 0.36/4.0 (8.9% complete) lr: 0.004889
2019-04-06 02:49:19,210 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 50/236 Epoch: 0.37/4.0 (9.1% complete) lr: 0.004862
2019-04-06 02:54:48,616 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 51/236 Epoch: 0.38/4.0 (9.4% complete) lr: 0.004835
2019-04-06 03:00:17,559 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 52/236 Epoch: 0.38/4.0 (9.6% complete) lr: 0.004808
2019-04-06 03:05:46,016 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 53/236 Epoch: 0.39/4.0 (9.9% complete) lr: 0.004782
2019-04-06 03:11:14,678 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 54/236 Epoch: 0.40/4.0 (10.1% complete) lr: 0.004755
2019-04-06 03:16:43,340 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 55/236 Epoch: 0.41/4.0 (10.3% complete) lr: 0.004729
2019-04-06 03:22:12,294 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 56/236 Epoch: 0.42/4.0 (10.6% complete) lr: 0.004703
2019-04-06 03:27:40,942 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 57/236 Epoch: 0.43/4.0 (10.8% complete) lr: 0.004677
2019-04-06 03:33:09,954 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 58/236 Epoch: 0.44/4.0 (11.1% complete) lr: 0.004651
2019-04-06 03:38:38,395 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 59/236 Epoch: 0.45/4.0 (11.3% complete) lr: 0.004626
2019-04-06 03:44:07,010 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 60/236 Epoch: 0.46/4.0 (11.5% complete) lr: 0.005750
2019-04-06 03:52:04,680 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 61/236 Epoch: 0.47/4.0 (11.8% complete) lr: 0.005710
2019-04-06 03:59:55,592 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 62/236 Epoch: 0.49/4.0 (12.1% complete) lr: 0.005671
2019-04-06 04:07:45,841 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 63/236 Epoch: 0.50/4.0 (12.4% complete) lr: 0.005632
2019-04-06 04:15:36,356 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 64/236 Epoch: 0.51/4.0 (12.7% complete) lr: 0.005593
2019-04-06 04:23:21,298 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 65/236 Epoch: 0.52/4.0 (13.0% complete) lr: 0.005555
2019-04-06 04:31:06,630 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 66/236 Epoch: 0.53/4.0 (13.3% complete) lr: 0.005516
2019-04-06 04:38:56,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 67/236 Epoch: 0.55/4.0 (13.6% complete) lr: 0.005478
2019-04-06 04:46:47,735 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 68/236 Epoch: 0.56/4.0 (13.9% complete) lr: 0.005440
2019-04-06 04:54:37,933 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 69/236 Epoch: 0.57/4.0 (14.2% complete) lr: 0.005403
2019-04-06 05:02:22,998 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 70/236 Epoch: 0.58/4.0 (14.5% complete) lr: 0.005366
2019-04-06 05:10:13,947 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 71/236 Epoch: 0.59/4.0 (14.8% complete) lr: 0.005329
2019-04-06 05:18:04,563 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 72/236 Epoch: 0.61/4.0 (15.1% complete) lr: 0.005292
2019-04-06 05:25:54,800 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 73/236 Epoch: 0.62/4.0 (15.4% complete) lr: 0.005256
2019-04-06 05:33:45,425 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 74/236 Epoch: 0.63/4.0 (15.7% complete) lr: 0.005219
2019-04-06 05:41:36,070 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 75/236 Epoch: 0.64/4.0 (16.0% complete) lr: 0.005183
2019-04-06 05:49:20,220 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 76/236 Epoch: 0.65/4.0 (16.3% complete) lr: 0.005148
2019-04-06 05:57:09,897 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 77/236 Epoch: 0.67/4.0 (16.6% complete) lr: 0.005112
2019-04-06 06:04:54,356 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 78/236 Epoch: 0.68/4.0 (16.9% complete) lr: 0.005077
2019-04-06 06:12:39,038 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 79/236 Epoch: 0.69/4.0 (17.2% complete) lr: 0.005042
2019-04-06 06:20:23,535 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 80/236 Epoch: 0.70/4.0 (17.5% complete) lr: 0.005007
2019-04-06 06:28:19,924 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 81/236 Epoch: 0.71/4.0 (17.8% complete) lr: 0.004973
2019-04-06 06:36:04,929 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 82/236 Epoch: 0.73/4.0 (18.1% complete) lr: 0.004938
2019-04-06 06:43:49,329 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 83/236 Epoch: 0.74/4.0 (18.4% complete) lr: 0.005885
2019-04-06 06:51:58,149 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 84/236 Epoch: 0.75/4.0 (18.8% complete) lr: 0.005836
2019-04-06 07:00:11,816 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 85/236 Epoch: 0.77/4.0 (19.2% complete) lr: 0.005788
2019-04-06 07:08:24,420 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 86/236 Epoch: 0.78/4.0 (19.5% complete) lr: 0.005740
2019-04-06 07:16:20,554 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 87/236 Epoch: 0.80/4.0 (19.9% complete) lr: 0.005693
2019-04-06 07:24:17,048 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 88/236 Epoch: 0.81/4.0 (20.3% complete) lr: 0.005646
2019-04-06 07:32:02,517 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 89/236 Epoch: 0.82/4.0 (20.6% complete) lr: 0.005599
2019-04-06 07:39:58,485 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 90/236 Epoch: 0.84/4.0 (21.0% complete) lr: 0.005553
2019-04-06 07:47:55,532 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 91/236 Epoch: 0.85/4.0 (21.3% complete) lr: 0.005507
2019-04-06 07:55:53,403 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 92/236 Epoch: 0.87/4.0 (21.7% complete) lr: 0.005461
2019-04-06 08:03:51,507 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 93/236 Epoch: 0.88/4.0 (22.1% complete) lr: 0.005416
2019-04-06 08:11:36,471 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 94/236 Epoch: 0.90/4.0 (22.4% complete) lr: 0.005371
2019-04-06 08:19:31,173 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 95/236 Epoch: 0.91/4.0 (22.8% complete) lr: 0.005327
2019-04-06 08:27:27,606 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 96/236 Epoch: 0.93/4.0 (23.1% complete) lr: 0.005283
2019-04-06 08:35:25,060 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 97/236 Epoch: 0.94/4.0 (23.5% complete) lr: 0.005239
2019-04-06 08:43:23,162 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 98/236 Epoch: 0.95/4.0 (23.9% complete) lr: 0.005196
2019-04-06 08:51:21,322 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 99/236 Epoch: 0.97/4.0 (24.2% complete) lr: 0.005153
2019-04-06 08:59:21,091 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 100/236 Epoch: 0.98/4.0 (24.6% complete) lr: 0.005110
2019-04-06 09:07:24,573 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 101/236 Epoch: 1.00/4.0 (24.9% complete) lr: 0.005068
2019-04-06 09:15:20,315 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 102/236 Epoch: 1.01/4.0 (25.3% complete) lr: 0.005026
2019-04-06 09:23:15,277 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 103/236 Epoch: 1.03/4.0 (25.7% complete) lr: 0.004985
2019-04-06 09:31:06,464 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 104/236 Epoch: 1.04/4.0 (26.0% complete) lr: 0.004943
2019-04-06 09:39:05,912 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 105/236 Epoch: 1.06/4.0 (26.4% complete) lr: 0.004903
2019-04-06 09:47:15,058 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 106/236 Epoch: 1.07/4.0 (26.7% complete) lr: 0.004862
2019-04-06 09:55:14,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 107/236 Epoch: 1.08/4.0 (27.1% complete) lr: 0.005625
2019-04-06 10:05:32,506 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 108/236 Epoch: 1.10/4.0 (27.5% complete) lr: 0.005571
2019-04-06 10:15:51,075 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 109/236 Epoch: 1.12/4.0 (27.9% complete) lr: 0.005517
2019-04-06 10:26:10,157 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 110/236 Epoch: 1.13/4.0 (28.4% complete) lr: 0.005464
2019-04-06 10:36:29,603 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 111/236 Epoch: 1.15/4.0 (28.8% complete) lr: 0.005412
2019-04-06 10:46:48,151 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 112/236 Epoch: 1.17/4.0 (29.2% complete) lr: 0.005359
2019-04-06 10:57:06,747 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 113/236 Epoch: 1.19/4.0 (29.6% complete) lr: 0.005308
2019-04-06 11:07:25,161 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 114/236 Epoch: 1.20/4.0 (30.0% complete) lr: 0.005257
2019-04-06 11:17:44,240 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 115/236 Epoch: 1.22/4.0 (30.5% complete) lr: 0.005206
2019-04-06 11:28:03,277 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 116/236 Epoch: 1.24/4.0 (30.9% complete) lr: 0.005156
2019-04-06 11:38:22,045 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 117/236 Epoch: 1.25/4.0 (31.3% complete) lr: 0.005106
2019-04-06 11:48:40,206 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 118/236 Epoch: 1.27/4.0 (31.7% complete) lr: 0.005057
2019-04-06 11:58:58,844 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 119/236 Epoch: 1.29/4.0 (32.2% complete) lr: 0.005008
2019-04-06 12:09:17,135 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 120/236 Epoch: 1.30/4.0 (32.6% complete) lr: 0.004960
2019-04-06 12:19:52,856 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 121/236 Epoch: 1.32/4.0 (33.0% complete) lr: 0.004912
2019-04-06 12:30:11,419 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 122/236 Epoch: 1.34/4.0 (33.4% complete) lr: 0.004865
2019-04-06 12:40:30,226 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 123/236 Epoch: 1.35/4.0 (33.8% complete) lr: 0.004818
2019-04-06 12:50:49,282 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 124/236 Epoch: 1.37/4.0 (34.3% complete) lr: 0.004771
2019-04-06 13:01:07,761 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 125/236 Epoch: 1.39/4.0 (34.7% complete) lr: 0.004725
2019-04-06 13:11:26,513 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 126/236 Epoch: 1.40/4.0 (35.1% complete) lr: 0.004680
2019-04-06 13:21:44,917 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 127/236 Epoch: 1.42/4.0 (35.5% complete) lr: 0.004635
2019-04-06 13:32:03,987 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 128/236 Epoch: 1.44/4.0 (35.9% complete) lr: 0.004590
2019-04-06 13:42:22,224 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 129/236 Epoch: 1.45/4.0 (36.4% complete) lr: 0.004546
2019-04-06 13:52:40,895 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 130/236 Epoch: 1.47/4.0 (36.8% complete) lr: 0.004502
2019-04-06 14:02:59,529 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 131/236 Epoch: 1.49/4.0 (37.2% complete) lr: 0.005095
2019-04-06 14:13:33,325 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 132/236 Epoch: 1.51/4.0 (37.7% complete) lr: 0.005039
2019-04-06 14:24:11,763 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 133/236 Epoch: 1.53/4.0 (38.2% complete) lr: 0.004984
2019-04-06 14:34:52,974 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 134/236 Epoch: 1.55/4.0 (38.6% complete) lr: 0.004929
2019-04-06 14:45:40,241 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 135/236 Epoch: 1.56/4.0 (39.1% complete) lr: 0.004875
2019-04-06 14:56:29,425 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 136/236 Epoch: 1.58/4.0 (39.6% complete) lr: 0.004821
2019-04-06 15:07:21,115 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 137/236 Epoch: 1.60/4.0 (40.1% complete) lr: 0.004768
2019-04-06 15:18:06,794 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 138/236 Epoch: 1.62/4.0 (40.6% complete) lr: 0.004716
2019-04-06 15:28:53,828 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 139/236 Epoch: 1.64/4.0 (41.0% complete) lr: 0.004664
2019-04-06 15:39:33,290 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 140/236 Epoch: 1.66/4.0 (41.5% complete) lr: 0.004612
2019-04-06 15:50:46,043 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 141/236 Epoch: 1.68/4.0 (42.0% complete) lr: 0.004562
2019-04-06 16:01:23,675 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 142/236 Epoch: 1.70/4.0 (42.5% complete) lr: 0.004511
2019-04-06 16:12:03,797 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 143/236 Epoch: 1.72/4.0 (43.0% complete) lr: 0.004462
2019-04-06 16:22:48,814 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 144/236 Epoch: 1.74/4.0 (43.4% complete) lr: 0.004413
2019-04-06 16:33:35,533 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 145/236 Epoch: 1.76/4.0 (43.9% complete) lr: 0.004364
2019-04-06 16:44:44,469 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 146/236 Epoch: 1.78/4.0 (44.4% complete) lr: 0.004316
2019-04-06 16:55:44,284 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 147/236 Epoch: 1.80/4.0 (44.9% complete) lr: 0.004268
2019-04-06 17:06:25,686 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 148/236 Epoch: 1.81/4.0 (45.4% complete) lr: 0.004221
2019-04-06 17:17:21,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 149/236 Epoch: 1.83/4.0 (45.9% complete) lr: 0.004175
2019-04-06 17:28:01,171 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 150/236 Epoch: 1.85/4.0 (46.3% complete) lr: 0.004129
2019-04-06 17:38:42,093 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 151/236 Epoch: 1.87/4.0 (46.8% complete) lr: 0.004083
2019-04-06 17:49:38,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 152/236 Epoch: 1.89/4.0 (47.3% complete) lr: 0.004039
2019-04-06 18:00:24,808 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 153/236 Epoch: 1.91/4.0 (47.8% complete) lr: 0.003994
2019-04-06 18:11:20,365 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 154/236 Epoch: 1.93/4.0 (48.3% complete) lr: 0.003950
2019-04-06 18:22:13,062 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 155/236 Epoch: 1.95/4.0 (48.7% complete) lr: 0.004395
2019-04-06 18:35:07,483 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 156/236 Epoch: 1.97/4.0 (49.3% complete) lr: 0.004341
2019-04-06 18:48:01,121 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 157/236 Epoch: 1.99/4.0 (49.8% complete) lr: 0.004287
2019-04-06 19:00:54,746 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 158/236 Epoch: 2.01/4.0 (50.4% complete) lr: 0.004234
2019-04-06 19:13:48,476 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 159/236 Epoch: 2.04/4.0 (50.9% complete) lr: 0.004181
2019-04-06 19:26:41,752 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 160/236 Epoch: 2.06/4.0 (51.4% complete) lr: 0.004130
2019-04-06 19:39:57,499 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 161/236 Epoch: 2.08/4.0 (52.0% complete) lr: 0.004079
2019-04-06 19:52:50,774 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 162/236 Epoch: 2.10/4.0 (52.5% complete) lr: 0.004028
2019-04-06 20:05:44,245 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 163/236 Epoch: 2.12/4.0 (53.1% complete) lr: 0.003978
2019-04-06 20:18:38,084 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 164/236 Epoch: 2.14/4.0 (53.6% complete) lr: 0.003929
2019-04-06 20:31:31,682 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 165/236 Epoch: 2.17/4.0 (54.1% complete) lr: 0.003880
2019-04-06 20:44:25,145 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 166/236 Epoch: 2.19/4.0 (54.7% complete) lr: 0.003832
2019-04-06 20:57:19,088 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 167/236 Epoch: 2.21/4.0 (55.2% complete) lr: 0.003785
2019-04-06 21:10:13,384 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 168/236 Epoch: 2.23/4.0 (55.8% complete) lr: 0.003738
2019-04-06 21:23:07,873 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 169/236 Epoch: 2.25/4.0 (56.3% complete) lr: 0.003692
2019-04-06 21:36:01,472 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 170/236 Epoch: 2.27/4.0 (56.9% complete) lr: 0.003646
2019-04-06 21:48:55,328 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 171/236 Epoch: 2.30/4.0 (57.4% complete) lr: 0.003601
2019-04-06 22:01:48,904 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 172/236 Epoch: 2.32/4.0 (57.9% complete) lr: 0.003556
2019-04-06 22:14:42,666 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 173/236 Epoch: 2.34/4.0 (58.5% complete) lr: 0.003512
2019-04-06 22:27:36,168 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 174/236 Epoch: 2.36/4.0 (59.0% complete) lr: 0.003469
2019-04-06 22:40:30,298 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 175/236 Epoch: 2.38/4.0 (59.6% complete) lr: 0.003426
2019-04-06 22:53:23,611 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 176/236 Epoch: 2.40/4.0 (60.1% complete) lr: 0.003384
2019-04-06 23:06:16,792 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 177/236 Epoch: 2.43/4.0 (60.6% complete) lr: 0.003342
2019-04-06 23:19:10,594 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 178/236 Epoch: 2.45/4.0 (61.2% complete) lr: 0.003667
2019-04-06 23:32:11,121 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 179/236 Epoch: 2.47/4.0 (61.8% complete) lr: 0.003617
2019-04-06 23:45:10,901 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 180/236 Epoch: 2.50/4.0 (62.4% complete) lr: 0.003567
2019-04-06 23:58:28,143 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 181/236 Epoch: 2.52/4.0 (63.0% complete) lr: 0.003518
2019-04-07 00:11:23,083 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 182/236 Epoch: 2.54/4.0 (63.6% complete) lr: 0.003470
2019-04-07 00:24:18,207 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 183/236 Epoch: 2.57/4.0 (64.2% complete) lr: 0.003422
2019-04-07 00:37:13,343 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 184/236 Epoch: 2.59/4.0 (64.8% complete) lr: 0.003375
2019-04-07 00:50:08,378 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 185/236 Epoch: 2.62/4.0 (65.4% complete) lr: 0.003328
2019-04-07 01:03:03,127 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 186/236 Epoch: 2.64/4.0 (66.0% complete) lr: 0.003283
2019-04-07 01:15:58,134 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 187/236 Epoch: 2.66/4.0 (66.6% complete) lr: 0.003238
2019-04-07 01:28:53,279 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 188/236 Epoch: 2.69/4.0 (67.2% complete) lr: 0.003193
2019-04-07 01:41:48,413 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 189/236 Epoch: 2.71/4.0 (67.8% complete) lr: 0.003149
2019-04-07 01:54:43,938 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 190/236 Epoch: 2.74/4.0 (68.4% complete) lr: 0.003106
2019-04-07 02:07:39,153 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 191/236 Epoch: 2.76/4.0 (69.0% complete) lr: 0.003063
2019-04-07 02:20:34,106 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 192/236 Epoch: 2.78/4.0 (69.6% complete) lr: 0.003021
2019-04-07 02:33:29,112 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 193/236 Epoch: 2.81/4.0 (70.2% complete) lr: 0.002980
2019-04-07 02:46:23,764 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 194/236 Epoch: 2.83/4.0 (70.8% complete) lr: 0.002939
2019-04-07 02:59:19,275 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 195/236 Epoch: 2.86/4.0 (71.4% complete) lr: 0.002898
2019-04-07 03:12:14,709 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 196/236 Epoch: 2.88/4.0 (72.0% complete) lr: 0.002859
2019-04-07 03:25:09,664 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 197/236 Epoch: 2.90/4.0 (72.6% complete) lr: 0.002819
2019-04-07 03:38:04,778 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 198/236 Epoch: 2.93/4.0 (73.2% complete) lr: 0.002780
2019-04-07 03:50:59,600 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 199/236 Epoch: 2.95/4.0 (73.8% complete) lr: 0.002742
2019-04-07 04:03:54,327 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 200/236 Epoch: 2.98/4.0 (74.4% complete) lr: 0.002705
2019-04-07 04:17:11,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 201/236 Epoch: 3.00/4.0 (75.0% complete) lr: 0.002667
2019-04-07 04:30:05,815 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 202/236 Epoch: 3.02/4.0 (75.6% complete) lr: 0.002894
2019-04-07 04:45:33,919 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 203/236 Epoch: 3.05/4.0 (76.3% complete) lr: 0.002850
2019-04-07 05:01:02,689 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 204/236 Epoch: 3.08/4.0 (76.9% complete) lr: 0.002807
2019-04-07 05:16:30,969 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 205/236 Epoch: 3.10/4.0 (77.6% complete) lr: 0.002765
2019-04-07 05:31:59,749 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 206/236 Epoch: 3.13/4.0 (78.2% complete) lr: 0.002723
2019-04-07 05:47:28,094 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 207/236 Epoch: 3.16/4.0 (78.9% complete) lr: 0.002682
2019-04-07 06:02:56,898 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 208/236 Epoch: 3.18/4.0 (79.6% complete) lr: 0.002641
2019-04-07 06:18:25,132 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 209/236 Epoch: 3.21/4.0 (80.2% complete) lr: 0.002601
2019-04-07 06:33:53,694 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 210/236 Epoch: 3.24/4.0 (80.9% complete) lr: 0.002562
2019-04-07 06:49:22,680 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 211/236 Epoch: 3.26/4.0 (81.6% complete) lr: 0.002523
2019-04-07 07:04:50,841 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 212/236 Epoch: 3.29/4.0 (82.2% complete) lr: 0.002485
2019-04-07 07:20:18,889 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 213/236 Epoch: 3.31/4.0 (82.9% complete) lr: 0.002448
2019-04-07 07:35:47,688 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 214/236 Epoch: 3.34/4.0 (83.5% complete) lr: 0.002411
2019-04-07 07:51:15,928 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 215/236 Epoch: 3.37/4.0 (84.2% complete) lr: 0.002374
2019-04-07 08:06:44,152 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 216/236 Epoch: 3.39/4.0 (84.9% complete) lr: 0.002338
2019-04-07 08:22:12,565 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 217/236 Epoch: 3.42/4.0 (85.5% complete) lr: 0.002303
2019-04-07 08:37:41,115 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 218/236 Epoch: 3.45/4.0 (86.2% complete) lr: 0.002268
2019-04-07 08:53:09,536 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 219/236 Epoch: 3.47/4.0 (86.8% complete) lr: 0.002234
2019-04-07 09:08:38,521 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 220/236 Epoch: 3.50/4.0 (87.5% complete) lr: 0.002200
2019-04-07 09:24:29,832 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 221/236 Epoch: 3.53/4.0 (88.2% complete) lr: 0.002167
2019-04-07 09:39:58,290 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 222/236 Epoch: 3.55/4.0 (88.8% complete) lr: 0.002134
2019-04-07 09:55:27,495 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 223/236 Epoch: 3.58/4.0 (89.5% complete) lr: 0.002102
2019-04-07 10:10:56,395 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 224/236 Epoch: 3.61/4.0 (90.1% complete) lr: 0.002070
2019-04-07 10:26:24,934 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 225/236 Epoch: 3.63/4.0 (90.8% complete) lr: 0.002039
2019-04-07 10:41:53,553 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 226/236 Epoch: 3.66/4.0 (91.5% complete) lr: 0.002191
2019-04-07 10:57:23,848 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 227/236 Epoch: 3.69/4.0 (92.2% complete) lr: 0.002155
2019-04-07 11:13:00,463 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 228/236 Epoch: 3.72/4.0 (92.9% complete) lr: 0.002119
2019-04-07 11:28:52,988 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 229/236 Epoch: 3.75/4.0 (93.6% complete) lr: 0.002084
2019-04-07 11:44:27,803 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 230/236 Epoch: 3.77/4.0 (94.4% complete) lr: 0.002050
2019-04-07 12:00:03,155 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 231/236 Epoch: 3.80/4.0 (95.1% complete) lr: 0.002016
2019-04-07 12:15:32,951 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 232/236 Epoch: 3.83/4.0 (95.8% complete) lr: 0.001983
2019-04-07 12:31:04,365 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 233/236 Epoch: 3.86/4.0 (96.5% complete) lr: 0.001950
2019-04-07 12:47:06,198 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 234/236 Epoch: 3.89/4.0 (97.2% complete) lr: 0.001918
2019-04-07 13:03:29,878 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 235/236 Epoch: 3.92/4.0 (98.0% complete) lr: 0.001887
2019-04-07 13:19:59,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 236/236 Epoch: 3.95/4.0 (98.7% complete) lr: 0.001800
2019-04-07 13:35:54,253 [steps/nnet3/train_dnn.py:398 - train - INFO ] Doing final combination to produce final.mdl
2019-04-07 13:35:54,263 [steps/libs/nnet3/train/frame_level_objf/common.py:491 - combine_models - INFO ] Combining set([224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 218, 219, 220, 221, 222, 223]) models.
2019-04-07 13:36:50,887 [steps/nnet3/train_dnn.py:407 - train - INFO ] Getting average posterior for purposes of adjusting the priors.
2019-04-07 13:39:47,914 [steps/nnet3/train_dnn.py:418 - train - INFO ] Re-adjusting priors based on computed posteriors
2019-04-07 13:39:48,078 [steps/nnet3/train_dnn.py:428 - train - INFO ] Cleaning up the experiment directory exp/nnet3/tdnn_sp
steps/nnet2/remove_egs.sh: Finished deleting examples in exp/nnet3/tdnn_sp/egs
exp/nnet3/tdnn_sp: num-iters=237 nj=2..12 num-params=12.2M dim=43+100->3032 combine=-0.49->-0.49 (over 4) loglike:train/valid[157,236,combined]=(-0.54,-0.47,-0.47/-0.70,-0.69,-0.69) accuracy:train/valid[157,236,combined]=(0.820,0.837,0.838/0.777,0.784,0.784)
steps/nnet3/decode.sh --nj 40 --cmd run.pl --mem 24G --online-ivector-dir exp/nnet3/ivectors_dev exp/tri5a/graph data/dev_hires exp/nnet3/tdnn_sp/decode_dev
steps/nnet3/decode.sh: feature type is raw
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 24G --iter final exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,3,20) and mean=8.8
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_dev/log/analyze_lattice_depth_stats.log
score best paths
+ steps/score_kaldi.sh --cmd 'run.pl --mem 24G' data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/score_kaldi.sh --cmd run.pl --mem 24G data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/score_kaldi.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ steps/scoring/score_kaldi_cer.sh --stage 2 --cmd 'run.pl --mem 24G' data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/scoring/score_kaldi_cer.sh --stage 2 --cmd run.pl --mem 24G data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/scoring/score_kaldi_cer.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ echo 'local/score.sh: Done'
local/score.sh: Done
score confidence and timing with sclite
Decoding done.
steps/nnet3/decode.sh --nj 20 --cmd run.pl --mem 24G --online-ivector-dir exp/nnet3/ivectors_test exp/tri5a/graph data/test_hires exp/nnet3/tdnn_sp/decode_test
steps/nnet3/decode.sh: feature type is raw
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 24G --iter final exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,4,26) and mean=11.4
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_test/log/analyze_lattice_depth_stats.log
score best paths
+ steps/score_kaldi.sh --cmd 'run.pl --mem 24G' data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/score_kaldi.sh --cmd run.pl --mem 24G data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/score_kaldi.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ steps/scoring/score_kaldi_cer.sh --stage 2 --cmd 'run.pl --mem 24G' data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/scoring/score_kaldi_cer.sh --stage 2 --cmd run.pl --mem 24G data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/scoring/score_kaldi_cer.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ echo 'local/score.sh: Done'
local/score.sh: Done
score confidence and timing with sclite
Decoding done.
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw
扫一扫
1727
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
