【小伟哥AI之路】Rasa之训练集与验证集评测指标及问题

本文主题：Rasa之训练集与验证集评测指标

在刚刚接触nlp自然语言处理这块，调试中，想到需要有测试集进行协调对比。

自动拆分nlu数据为训练集和测试集

rasa data split nlu

 测试集数据验证制定nlu模型进行测试

rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz

 If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag --cross-validation:

如果不想创建单独的测试集，仍然可以使用交叉验证来估计模型的泛化程度。--cross-validation 

此过程不会生成模型，会直接输入结果

rasa test nlu -u data/nlu.md --config config.yml --cross-validation

扩展参数 -f 为层叠划分，系统默认为5层（怎么理解？每次把数据分成5份，为1/5测试集，依次作为测试推理5次，取结果）

rasa test nlu -u data/nlu.md --config config.yml --cross-validation -f 3

扩展参数列表说明：

usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to a trained Rasa model. If a directory is
                        specified, it will use the latest model in this
                        directory. (default: models)
  -u NLU, --nlu NLU     File or folder containing your NLU data. (default:
                        data)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --successes           If set successful predictions (intent and entities)
                        will be written to a file. (default: False)
  --no-errors           If set incorrect predictions (intent and entities)
                        will NOT be written to a file. (default: False)
  --histogram HISTOGRAM
                        Output path for the confidence histogram. (default:
                        hist.png)
  --confmat CONFMAT     Output path for the confusion matrix plot. (default:
                        confmat.png)
  -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
                        Model configuration file. If a single file is passed
                        and cross validation mode is chosen, cross-validation
                        is performed, if multiple configs or a folder of
                        configs are passed, models will be trained and
                        compared directly. (default: None)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Cross Validation:
  --cross-validation    Switch on cross validation mode. Any provided model
                        will be ignored. (default: False)
  -f FOLDS, --folds FOLDS
                        Number of cross validation folds (cross validation
                        only). (default: 5)

Comparison Mode:
  -r RUNS, --runs RUNS  Number of comparison runs to make. (default: 3)
  -p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
                        Percentages of training data to exclude during
                        comparison. (default: [0, 25, 50, 75])

运行成功如下指标：

2019-12-31 19:26:46 INFO     rasa.nlu.test  - Accumulated test folds entity evaluation results:
2019-12-31 19:26:47 INFO     rasa.nlu.test  - Evaluation for entity extractor: CRFEntityExtractor 
2019-12-31 19:26:47 INFO     rasa.nlu.test  - Classification report for 'CRFEntityExtractor' saved to 'results/CRFEntityExtractor_report.json'.
2019-12-31 19:26:47 INFO     rasa.nlu.test  - Incorrect entity predictions saved to results/CRFEntityExtractor_errors.json.
2019-12-31 19:26:47 INFO     rasa.test  - CV evaluation (n=2)
2019-12-31 19:26:47 INFO     rasa.test  - Intent evaluation results
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train Accuracy: 1.000 (0.000)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train F1-score: 1.000 (0.000)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train Precision: 1.000 (0.000)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test Accuracy: 0.849 (0.019)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test F1-score: 0.842 (0.020)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test Precision: 0.877 (0.008)
2019-12-31 19:26:47 INFO     rasa.test  - Entity evaluation results
2019-12-31 19:26:47 INFO     rasa.nlu.test  - Entity extractor: CRFEntityExtractor
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train Accuracy: 0.997 (0.001)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train F1-score: 0.996 (0.002)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - train Precision: 0.999 (0.001)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - Entity extractor: CRFEntityExtractor
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test Accuracy: 0.990 (0.002)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test F1-score: 0.987 (0.002)
2019-12-31 19:26:47 INFO     rasa.nlu.test  - test Precision: 0.997 (0.003)

 

Python在Mac上报错:RuntimeError: Python is not installed as a framework.解决办法

在mac上使用virtualenv虚拟机，import matplotlib或者import seaborn时报错：

解决方法：
RuntimeError: Python is not installed as a framework. The
Mac OS X backend will not be able to function correctly if Python
is not installed as a framework. See the Python documentation for
more information on installing Python as a framework on Mac OS X.
Please either reinstall Python as a framework, or try one of the other 
backends.
 在虚拟机中或者命令行状态下输入：

 vim ~/.matplotlib/matplotlibrc

 创建一个名为matplotlib的文件，按i 进入编辑模式，输入

 backend: TkAgg

 按esc退出编辑模式，输入:wq 保存并退出，重启终端进入python即可。

如果遇到以下提示：E212：Can't open file for writing

则需要管理员权限使用vim，输入

sudo vim ~/.matplotlib/matplotlibrc

 

参考官方文档：

https://rasa.com/docs/rasa/user-guide/evaluating-models/

习惯看官网，英文不是母语怎么办。

找个女朋友，帮忙翻一下