【计算机科学】【2016.09】时间序列的训练策略：过滤学习与强化学习

本文为美国佐治亚理工学院（作者：Arun Venkatraman）的博士论文，共97页。

数据驱动的时间序列建模方法在经济学的市场预测和机器人系统仿真等各种应用中都很重要。然而，传统的为i.i.d.数据设计的有监督机器学习技术在这些序列问题上常常表现不佳。本文提出无论是对于预测、滤波还是强化学习，直接训练递归预测过程本身，而不是建立生成概率模型，可以有效地实现时间序列和序列预测。

为此，我们介绍了一种用于学习时间序列模型的新训练算法，Data asDemonstrator (DaD)，该算法在理论和实验上改进了递归神经网络、核回归器和随机森林等模型的多步预测性能。此外，实验结果表明DaD可以加速基于模型的强化学习。接下来，我们展示了隐态时间序列模型可以以一种监督的方式有效地学习，其中的充分状态参数化可能是未知的。我们的预测状态推理机（PSIMs）方法，通过识别重复的隐藏状态作为对未来观测统计数据的预测信念，直接优化DaD风格的训练过程，而不会陷入局部最优的推理性能。可观测量预测作为构建AI系统的通用语言，是我们学习框架的基础。我们通过这种总体思想提出了三种扩展方案来适应各种问题。第一个目标是提高更复杂递归神经网络的训练时间和性能；第二个目标将PSIM框架扩展到受控的动态系统；第三个目标是训练用于强化学习问题的递归架构。

Data driven approaches to modelingtime-series are important in a variety of applications from market predictionin economics to the simulation of robotic systems. However, traditionalsupervised machine learning techniques designed for i.i.d. data often performpoorly on these sequential problems. This thesis proposes that time series andsequential prediction, whether for forecasting, filtering, or reinforcementlearning, can be effectively achieved by directly training recurrent predictionprocedures rather then building generative probabilistic models. To this end,we introduce a new training algorithm for learned time-series models, Data asDemonstrator (DaD), that theoretically and empirically improves multi-stepprediction performance on model classes such as recurrent neural networks,kernel regressors, and random forests. Additionally, experimental resultsindicate that DaD can accelerate model-based reinforcement learning. We nextshow that latent-state time-series models, where a sufficient stateparametrization may be unknown, can be learned effectively in a supervised way.Our approach, Predictive State Inference Machines (PSIMs), directly optimizes {through a DaD-style training procedure { the inference performance withoutlocal optima by identifying the recurrent hidden state as a predictive beliefover statistics of future observations. Fundamental to our learning frameworkis that the prediction of observable quantities is a lingua franca for buildingAI systems. We propose three extensions that leverage this general idea andadapt it to a variety of problems. The first aims to improve the training timeand performance of more sophisticated recurrent neural networks. The secondextends the PSIM framework to controlled dynamical systems. The third looks totrain recurrent architectures for reinforcement learning problems.

1 引言
2 项目背景
2.1 时间序列建模
2.2 动态系统模型的应用
3 本文工作内容
3.1 Data asDemonstrator : 多步预测迭代时间序列模型的改进训练方法
3.2 改进的DaD动态控制
3.3 非参数过滤学习：预测状态推理机
3.4 基于征兆的扩展PSIM
4 未来工作展望
4.1 基于PSIM的RNN结构化训练
4.2 预测状态推理机的建模行为
4.3 无模型RL的递归模型
5 结论与时间表

下载英文原文地址：

http://page5.dfpan.com/fs/7lcj6221b29156b6fc2/

更多精彩文章请关注微信号：