Kaldi单步完美运行AIShell v1 S5之五：DNN (chain)

致谢

感谢AIShell在商业化道路上的探索。期待着v3的到来。

机器配置

sv@HP:~$ sudo lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

sv@HP:~$ cat /proc/cpuinfo | grep model\ name
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
model name	: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
sv@HP:~$ cat /proc/meminfo | grep MemTotal
MemTotal:       16321360 kB
sv@HP:~$ lspci | grep 'VGA'
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)

问题：显卡设备老旧，一个GPU，想跑tdnn模型，如何破？

**解答：**
将num-jobs-initial和num-jobs-final都设为1，将epochs改为2或者3。
GPU设为独占。
sv@HP:~/lkaldi/egs/aishell/s5$ sudo nvidia-smi -c 3
[sudo] password for sv: 
Set compute mode to EXCLUSIVE_PROCESS for GPU 00000000:01:00.0.
All done.
sv@HP:~/lkaldi/egs/aishell/s5$ sudo nvidia-smi
Wed Jan 16 10:31:58 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
| 27%   31C    P8     7W / 151W |    225MiB /  8116MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1432      G   /usr/lib/xorg/Xorg                           125MiB |
|    0      1645      G   /usr/bin/gnome-shell                          94MiB |
|    0      2622      G   /opt/firefox/firefox-bin                       3MiB |
+-----------------------------------------------------------------------------+

Kaldi单步完美运行AIShell v1 S5之五：DNN (chain)

终篇。Chain Model的结果可以线上、实时，才有独立的商用价值。

第14部分：DNN Chain Model

先看结果。DNN nnet3错词率降到8.68%。Chain降到7.72%。

sv@HP:~/lkaldi/egs/aishell/s5$ for x in exp/ */decode_test; do [ -d $x ] && grep WER $x/cer_* | utils/best_wer.sh; done 2>/dev/null

%WER 36.59 [ 38335 / 104765, 849 ins, 3183 del, 34303 sub ] exp/mono/decode_test/cer_10_0.0
%WER 18.83 [ 19727 / 104765, 971 ins, 1161 del, 17595 sub ] exp/tri1/decode_test/cer_13_0.5
%WER 18.79 [ 19684 / 104765, 957 ins, 1142 del, 17585 sub ] exp/tri2/decode_test/cer_14_0.5
%WER 16.84 [ 17643 / 104765, 791 ins, 991 del, 15861 sub ] exp/tri3a/decode_test/cer_14_0.5
%WER 13.63 [ 14277 / 104765, 762 ins, 639 del, 12876 sub ] exp/tri4a/decode_test/cer_13_0.5
%WER 8.68 [ 9097 / 104765, 355 ins, 464 del, 8278 sub ] exp/nnet3/tdnn_sp/decode_test/cer_14_1.0
%WER 7.72 [ 8087 / 104765, 364 ins, 552 del, 7171 sub ] exp/chain/tdnn_1a_sp/decode_test/cer_11_0.5

第12部分：Chain训练、解码、校准

sv@HP:~/lkaldi/egs/aishell/s5$ local/chain/run_tdnn.sh
local/chain/run_tdnn.sh 
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
utils/data/perturb_data_dir_speed_3way.sh: data/train_sp/feats.scp already exists: refusing to run this (please delete data/train_sp/feats.scp if you want this to run)
sv@HP:~/lkaldi/egs/aishell/s5$ local/chain/run_tdnn.sh
local/chain/run_tdnn.sh 
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train/utt2dur already exists with the expected length.  We won't recompute it.
utils/data/get_reco2dur.sh: data/train/reco2dur already exists with the expected length.  We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train, in data/train_sp_speed0.</