Reinforcement Learning Exercise 5.5

最新推荐文章于 2019-10-03 12:31:59 发布
最新推荐文章于 2019-10-03 12:31:59 发布
阅读量1.1k 收藏
点赞数
分类专栏： reinforcement learning 文章标签： reinforcement learning
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ballade2012/article/details/98348253
版权
在含有单一非终态状态和单一行动的MDP中，概率p返回非终态，1-p进入终态，奖励始终为1，γ=1。在一个持续10步并得到回报10的实验中，首次访问和每次访问状态下价值估计分别为p和1-p/(p^10-1)。
摘要由CSDN通过智能技术生成

Exercise 5.5 Consider an MDP with a single nonterminal state and a single action that transitions back to the nonterminal state with probability p p p and transitions to the terminal state with probability 1 − p 1-p 1−p. Let the reward be + 1 +1 +1 on all transitions, and let γ = 1 \gamma=1 γ=1. Suppose you observe one episode that lasts 10 steps, with a return of 10. What are the first-visit and every-visit estimators of the value of the nonterminal state?

For the first-visit estimator, only the first visit of a state is considered. So:
V ( S n o n t e r m i n a l ) = G ( S 0 ) = 1 ⋅ p + 0 ⋅ ( 1 − p ) = p \begin{aligned} V(S_{nonterminal}) &= G(S_0) \\ &= 1 \cdot p + 0 \cdot (1-p) \\ &= p \end{aligned} V(S

最低0.47元/天 解锁文章
YeXiang\^-^/
关注
专栏目录
Reinforcement Learning Exercise 3.19
ballade2012的博客
04-09 675
Exercise 3.19 The value of an action, qπ(s,a)q_\pi(s, a)qπ​(s,a), depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small back...
Reinforcement Learning Exercise 3.15
ballade2012的博客
05-27 493
Exercise 3.15 In the gridworld example, rewards are positive for goals, negative for running into the edge of the world, and zero the rest of the time. Are the signs of these rewards important, or onl...
Reinforcement Learning Exercise 5.13
ballade2012的博客
09-10 287
Exercise 5.13 Show the steps to derive (5.14) from (5.12). ρt:T−1Rt+1=π(At∣St)b(At∣St)π(At+1∣St+1)b(At+1∣St+1)π(At+2∣St+2)b(At+2∣St+2)⋯π(AT−1∣ST−1)b(AT−1∣ST−1)Rt+1(5.12) \rho_{t:T-1}R_{t+1} = \frac{\p...
Reinforcement Learning Exercise 3.18
ballade2012的博客
04-07 395
Exercise 3.18 υπ(s)=Eπ(Gt∣St=s)=∑a∈AEπ(Gt∣St=s,At=a)P(At=a∣St=s)∵P(At=a∣St=s)=π(a∣s)∴υπ(s)=∑a∈AEπ(Gt∣St=s,At=a)π(a∣s) \begin{aligned} \upsilon_\pi(s) &= \mathbb E_\pi ( G_t | S_t = s ) \\ &amp...
Reinforcement Learning exercise 3.13
ballade2012的博客
05-26 604
Exercise 3.13 Give an equation for qπq_\piqπ​ in terms of vπv_\pivπ​ and the four-argument ppp. First, we need to derive a formula from multiplication formula of probability theory: p(x∣y)=p(x,y)p(y)=...
Reinforcement Learning Exercise 3.24
ballade2012的博客
05-02 660
Exercise 3.24 Figure 3.5 gives the optimal value of the best state of the gridworld as 24.4, to one decimal place. Use your knowledge of the optimal policy and (3.8) to express this value symbolically...
Reinforcement learning an introduction中文pdf
08-25
Reinforcement learning an introduction中文pdf
Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions:强化学习的解决方案，简介
05-10
强化学习解决方案第二版（Richard S. Sutton的，Andrew G. Barto） 第12章已更新。 有关详细信息，请参见下面的日志。 那些正在使用它来完成您的作业的学生，​​请停止该作业。 本手册旨在为数百万没有官方指南或...
reinforcement-learning-an-introduction-chinese:《强化学习
04-30
本项目为《Reinforcement Learning: An Introduction》（第二版）中文翻译，旨在帮助喜欢 强化学习（Reinforcement Learning）的各位能更好的学习交流。 中文在线阅读地址：《》 英文原版地址： 翻译进度： 第二版...
Reinforcement Learning Exercises 6.1
ballade2012的博客
10-03 342
Exercise 6.1 If VVV changes during the episode, then (6.6) only holds approximately; what would the difference be between the two sides? Let VtV_tVt​ denote the array of state values used at time ttt ...
Reinforcement Learning: An Introduction 2nd solutions （第二版 答案）
04-09
sutton的强化学习第二版的答案，不确定是否为sutton亲自提供的，仅供参考，还是很有帮助的
Reinforcement Learning Exercise 4.9
ballade2012的博客
07-04 811
Exercise 4.9 (programming) Implement value iteration for the gambler’s problem and solve it for php_hph​ = 0.25 and php_hph​ = 0.55. In programming, you may find it convenient to introduce two dummy s...
Reinforcement Learning Chapter 5 Example 5.5
ballade2012的博客
08-05 180
In the book, example 5.5 shows the infinity variance of the ordinary importance sampling in a specific case. I tried this experiment in my computer and get a similar result. Unfortunately, my computer...
《reinforcement learning：an introduction》第二章《Multi-arm Bandits》总结
mmc2015的专栏
07-10 1517
由于组里新同学进来，需要带着他入门RL，选择从silver的课程开始。 对于我自己，增加一个仔细阅读《reinforcement learning：an introduction》的要求。 因为之前读的不太认真，这一次希望可以认真一点，将对应的知识点也做一个简单总结。 K-armed bandit problem:     Consider the followin
Reinforcement Learning Exercise 5.1 & 5.2
ballade2012的博客
08-03 580
Exercise 5.1 Consider the diagrams on the right in Figure 5.1. Why does the estimated value function jump up for the last two rows in the rear? Why does it drop off for the whole last row on the left?...
Reinforcement Learning Exercise 5.10
ballade2012的博客
08-10 238
Exercise 5.10 Derive the weighted-average update rule (5.8) from (5.7). Follow thepattern of the derivation of the unweighted rule (2.3)
强化学习系列（五）：蒙特卡罗方法（Monte Carlo)
热门推荐
LagrangeSK的博客
07-27 2万+
一、前言 在强化学习系列（四）：动态规划中，我们介绍了采用DP方法求解environment model 已知的MDP，本章将介绍一种无模型的强化学习方法——蒙特卡洛方法（Monte Carlo)。...
Reinforcement Learning Exercise 4.7
ballade2012的博客
07-04 1656
Exercise 4.7 (programming) Write a program for policy iteration and re-solve Jack’s car rental problem with the following changes. One of Jack’s employees at the first location rides a bus home each n...
Reinforcement Learning Exercise 4.4
ballade2012的博客
06-21 1372
Exercise 4.4 The policy iteration algorithm on page 80 has a subtle bug in that it may never terminate if the policy continually switches between two or more policies that are equally good. This is OK...
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值