4.1 Temporal Differential of one step

最新推荐文章于 2022-05-06 09:00:00 发布
最新推荐文章于 2022-05-06 09:00:00 发布
阅读量104 收藏
点赞数
分类专栏： 强化学习学习笔记 文章标签： python reinforcement learning
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/upr_rom/article/details/119698558
版权

目录

Definition

Prediction of TD(0)

Random Walk

On-policy TD Control (Sarsa)

Windy Gridworld

Code

Off-policy TD Control (Q-learning)

Result

Code

Expected Sarsa 

Result

Code 

Definition

v_{\pi_{t}}(s) \ \\ =\ E[ R_{t+1}+\gamma v_{ \pi_{t+1}} ] = \sum _a \pi(a|s) \sum _{r,s'}p(r,s'|s,a)(r+\gamma v(s')) =max_a \ \sum _{r,s'}p(r,s'|s,a)(r+\gamma v(s')) \\ =E[ G_t ] = average(G_t) \ in\ every\ s \\ \ =E[ R_{t+1}+\gamma G_{t+1} ]\approx average(R_{t+1}+\gamma v(s'))

q_{\pi_t}(s,a) \\=E[R_{t+1}+\gamma q_{\pi_{t+1}}(s,a)]=\sum _{r,s'}p(r,s'|s,a)(r+\sum _{a'}\pi(a'|s')\gamma q(s',a')) = \sum _{r,s'}p(r,s'|s,a)(r+\gamma max_{a'} \ q(s',a')) \\=E[G_t(s,a)]=average(R_{t+1}+\gamma G_{t+1}(s,a)) \\=E[R_{t+1}+\gamma G_{t+1}]=average(R_{t+1}+\gamma q(S_{t+1},A_{t+1}))

 Dynamic Programming uses the euqations in the second line and has to know the environmental dynamics ( dynamics of environemnt  can produce the chain between this state to the next state, but it's hard to know. Cons ) ( It uses the relationship between different states, which creates bootstrapping. Experience from other states can be used in this state. Pros )

Monte Carlo Method uses the equations in the third line and has to know the returns from entire results ( which can produce the relationship between different states when we do not know the environmental dynamics Pros, and dynamics of environment may not precise as the Monte Carlo) ( However, when the episodes are not fully visited and updated, the values can only be updates partically )

Temporal Differential (shorten as TD) is the combination of DP and MC. First, it could update from other states, which means the states from A episode(not appear in B episide) can use the experience from B episide, because the value table is commonly used.  It can avoid partially estimation from MC. Secondly, it uses the episodes from real world, we do not need the dynamics of programming. 

Prediction of TD(0)

loop until the error < MINIMUM_OF_LIMIT:

        v_last = v;

        loop in the episode:        

                Choose action from policy in state, then get the reward and next state from episode;

                v(s) = (1-\alpha )v(s)+\alpha (R+v(s'))

        error = maximum( |v - v_last| ) 

Random Walk

The result tells us TD is quiet greater than MC. 

##  settings

import math
import numpy as np
import random

# visualization 
import matplotlib 
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator
# INVARIANT 
# state

STATE_LEFT = 0;
A = 1 ;
B = 2;
C = 3;
D = 4;
E = 5;
STATE_RIGHT = 6;

STATE =[ A,B,C,D,E ]
TERMINAL_STATE = [STATE_LEFT,STATE_RIGHT];


# action

LEFT = -1;
RIGHT = 1;

ACTION = [LEFT,RIGHT];

# random policy 

random_policy = np.zeros((7,2),dtype = np.float);
for state in STATE:
	random_policy[state][LEFT] = 0.5;
	random_policy[state][RIGHT] = 0.5;

# discount 
gamma = 1; 
ALFA =[0.05,0.1,0.2];

# loop number
LOOP_EPISODE = 1000;

targets = [1.0/6,2.0/6,3.0/6,4.0/6,5.0/6]

# get reward 

def get_reward(next_state):
	return 1 if next_state==STATE_RIGHT else 0; 


# get action

def get_action(state,policy):
	p=[];
	for action in ACTION:
		p.append(policy[state][action]) ; 
	return np.random.choice(ACTION,p=p)
	


# Agent  class
class Agent_class():
	
	def __init__(self):
		self.state_set=[];
		self.reward_set=[];
		self.current_state = C;
		self.state_set.append(C);
	
	def finish_sampling(self,policy):
		while self.current_state in STATE:
			action = get_action( self.current_state,policy );
			self.current_state += action;
			
			self.state_set.append(self.current_state);
			self.reward_set.append(get_reward(self.current_state));


def get_rms(predictions,targets):
	return np.sqrt( ((predictions-targets)**2).mean() );




for alfa in ALFA:
	
	V_MC = np.zeros(7,dtype = np.float);
	V_TD = np.zeros(7,dtype = np.float);

	MC_RSM=[];
	TD_RSM=[];
		
	
	for every_loop in range(0,LOOP_EPISODE):
		
		agent = Agent_class();
		agent.finish_sampling(random_policy);
		
		for state in range(0,len(agent.state_set)-1):
			
			current_state = agent.state_set[state];
			next_state = agent.state_set[state+1];
			V_TD[current_state] = (1-alfa)*V_TD[ current_state ] + \
			alfa*( agent.reward_set[state] + gamma *V_TD[ next_state ]);
		
		G_t = agent.reward_set[-1];
		for i in range(1,len(agent.state_set)): 
			
			current_state = agent.state_set[-i-1];
			next_state = agent.state_set[-i];
			
			V_MC[current_state] = (1-alfa)*V_MC[current_state] + \
			alfa*gamma*G_t; 
			
		MC_RSM.append( get_rms(V_MC[1:6],targets) );
		TD_RSM.append( get_rms(V_TD[1:6],targets) );
		
		
	plt.plot(MC_RSM,'-',label='alfa='+str(alfa)+' MC');
	plt.plot(TD_RSM,'-',label='alfa='+str(alfa)+' TD');
	
plt.legend();

plt.show()

'''
plt.plot(V_MC[1:6],label='alfa='+str(alfa)+' MC')	
plt.plot(V_TD[1:6],label='alfa='+str(alfa)+' TD')
plt.legend();
plt.show()	


'''

 On-policy TD Control (Sarsa)

After every updation of every state in an episode, we could update the policy quickly rather than after every total episode in Monte Carlo. 

Windy Gridworld

State: every position in this grid, the terminal state is the goal position.

Action: left,right,up,down. there are some additional actions in the center of the gridworld.

reward: -1, until arrive at the terminal state.

policy evauluation: TD(0)

policy improvement: update after the iteration in every state

When soft-policy is applied (sigma=0.05,it's constant), we could get good result even optimal result sometimes. 

 When soft-policy is applied (sigma=0.05,it's decreasing from 0.05 to 0), we could get optimal result .

 

 Code

##  settings

import math
import numpy as np
import random

# visualization 
import matplotlib 
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator

# list copy
import copy


# INVARIANTS 

#state   (x,y) 

STATE_INITIAL = (0,3);
STATE_GOAL = (7,3)

WIDTH = 10;
HEIGHT = 7;


# action
LEFT = (-1,0);
RIGHT = (1,0);
UP = (0,1);
DOWN = (0,-1);

ACTION_INTENDED = [LEFT,RIGHT,UP,DOWN];

# policy pi(a|s) which start at random policy, and updated after every iteration (function)
policy_target={};  # policy[(x,y)][action]

dict_actions_policy={};
for action in ACTION_INTENDED:
	dict_actions_policy[action]=1/len(ACTION_INTENDED);

for x in range(0,WIDTH):
	for y in range(0,HEIGHT):	
		policy_target[(x,y)] = copy.deepcopy(dict_actions_policy)


# reward: q_s_a(STATE_GOAL)=0, reward in every step is -1. 
R = -1; 


# special part: windy_effect, which will change the action_intended into action_actual
# 	there are different influence on different x. 
# 	1 unit influence will produce additional up in gridworld.
ACTION_WIND_EFFECT = [0,0,0,1,1,1,2,2,1,0];


# parameters we set

# sigma-policy  
SIGMA = 0.05;
# number of episode
LOOP_EPISODE = 200;
# discounting parameter
gamma = 1;
# iteration step 
alfa = 0.5;


# FUNCTIONS

# FUNCTIONS in policy evaluation
def choose_action(state,policy):
	p=[];
	index=range(0,len(ACTION_INTENDED));
	for action in ACTION_INTENDED:
		p.append(policy[state][action]); 
	action_intended = ACTION_INTENDED[np.random.choice(index,p=p)];	
	return action_intended;

# actual action taken (update after choice of actions )
def get_actual_action(state,action_intended):
	x = action_intended[0]
	y = action_intended[1] + ACTION_WIND_EFFECT[ state[0] ];
	return ( (x,y) );


# state after the actual action taken
# the agent could not get out of the grid world.
def get_state_next(state,action):
	state_next = ( state[0]+action[0], state[1]+action[1] );
	x = max(state_next[0],0);
	x = min(x,WIDTH-1);
	
	y = max(state_next[1],0);
	y = min(y,HEIGHT-1);
	return (x,y);


# FUNCTIONS in policy improvement

# update after every iteration and use sigma-policy 
def policy_updation(state,Q_s_a,policy):
	
	Q=[];
	for action in ACTION_INTENDED:
		Q.append(Q_s_a[state][action]);
	action_max = ACTION_INTENDED[np.argmax(Q)];
	for action in ACTION_INTENDED:
		policy[state][action] = sigma/len(ACTION_INTENDED);
	policy[state][action_max] += 1 - sigma;


# CLASS 

# CLASS Agent 

class Agent_class():
	
	def __init__(self):
		
		# q_s_a[state][action]  initial value =0;
		self.Q_s_a={};
		self.policy = copy.deepcopy( policy_target );
		
		dict_q_action={};
		for action in ACTION_INTENDED:
			dict_q_action[action] = 0;
		for x in range(0,WIDTH):
			for y in range(0,HEIGHT):
				self.Q_s_a[(x,y)] = copy.deepcopy( dict_q_action );
		
		self.action_total_set=[];
		self.state_total_set=[];
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy)
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);
		


	def finish_journey(self):
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy)
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);
				
		# state set and action set
		action_set=[];
		state_set=[];
		
		action_set.append( self.current_intended_action );
		state_set.append( self.current_state );
			
		next_state = get_state_next(self.current_state,self.current_actual_action);
		while next_state != STATE_GOAL:
			# get the next action;
			next_action_intended = choose_action(next_state,self.policy);
			next_action_actual = get_actual_action(next_state,next_action_intended)
			# policy evaluation
			self.Q_s_a[self.current_state][self.current_intended_action] = (1-alfa)*\
			self.Q_s_a[self.current_state][self.current_intended_action] + alfa* \
			(R + self.Q_s_a[next_state][next_action_intended]);
			#policy improvement
			policy_updation(self.current_state,self.Q_s_a,self.policy)
			# update current state and action
			self.current_intended_action = next_action_intended;
			self.current_actual_action = next_action_actual;
			self.current_state = next_state;
			# append of state set and action set
			action_set.append( self.current_intended_action );
			state_set.append( self.current_state );
			# get next state
			next_state = get_state_next(self.current_state,self.current_actual_action);
			
		# update when next state is terminal state 
		state_set.append(next_state)
		self.Q_s_a[self.current_state][self.current_intended_action] = (1-alfa)*\
		self.Q_s_a[self.current_state][self.current_intended_action] + alfa* R;
		
		self.action_total_set.append(action_set);
		self.state_total_set.append(state_set);
		
	def show_step(self):
		x=[];
		y=[];
		for state in self.state_total_set[-1]:
			x.append(state[0]);
			y.append(state[1]);
		
		plt.plot(x,y,'->')
		plt.plot(STATE_GOAL[0],STATE_GOAL[1],'*')
		plt.plot(STATE_INITIAL[0],STATE_INITIAL[1],'*')
		plt.show();
	
	def show_step_number_of_every_episode(self):
		number=[];
		for i in range(0,len(self.action_total_set)):
			number.append(len(self.action_total_set[i]));
		plt.plot(number,'-*')
		plt.xlabel('number of episodes')
		plt.ylabel('number of steps')
		plt.show();
			
		
		

# main programming 

agent = Agent_class();

for every_loop in range(0,LOOP_EPISODE):
	#sigma = SIGMA*(LOOP_EPISODE-every_loop)/LOOP_EPISODE;
	sigma = SIGMA;
	agent.finish_journey();
agent.show_step();
agent.show_step_number_of_every_episode();

 Off-policy TD Control (Q-learning)

Result

When soft-policy is applied (sigma=0.05,constant), we could get good result . Because it will have some non-perfect actions.

When soft-policy is applied (sigma=0.05,it's decreasing from 0.05 to 0), we could get optimal result . But more episodes are needed than Sarsa.

 

 

 Code

##  settings

import math
import numpy as np
import random

# visualization 
import matplotlib 
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator

# list copy
import copy


# INVARIANTS 

#state   (x,y) 

STATE_INITIAL = (0,3);
STATE_GOAL = (7,3)

WIDTH = 10;
HEIGHT = 7;


# action
LEFT = (-1,0);
RIGHT = (1,0);
UP = (0,1);
DOWN = (0,-1);

ACTION_INTENDED = [LEFT,RIGHT,UP,DOWN];

# policy pi(a|s) which start at random policy, and updated after every iteration (function)
policy_target={};  # policy[(x,y)][action]

dict_actions_policy={};
for action in ACTION_INTENDED:
	dict_actions_policy[action]=1/len(ACTION_INTENDED);

for x in range(0,WIDTH):
	for y in range(0,HEIGHT):	
		policy_target[(x,y)] = copy.deepcopy(dict_actions_policy)


# reward: q_s_a(STATE_GOAL)=0, reward in every step is -1. 
R = -1; 


# special part: windy_effect, which will change the action_intended into action_actual
# 	there are different influence on different x. 
# 	1 unit influence will produce additional up in gridworld.
ACTION_WIND_EFFECT = [0,0,0,1,1,1,2,2,1,0];


# parameters we set

# sigma-policy  
SIGMA = 0.05;
# number of episode
LOOP_EPISODE = 100;
# discounting parameter
gamma = 1;
# iteration step 
alfa = 0.5;


# FUNCTIONS

# FUNCTIONS in policy evaluation
def choose_action(state,policy):
	p=[];
	index=range(0,len(ACTION_INTENDED));
	for action in ACTION_INTENDED:
		p.append(policy[state][action]); 
	action_intended = ACTION_INTENDED[np.random.choice(index,p=p)];	
	return action_intended;

def choose_max_action(state,Q_s_a):
	Q=[];
	choice=[];
	for action in ACTION_INTENDED:
		Q.append(Q_s_a[state][action]);
	for i in range(0,len(Q)):
		if Q[i]==max(Q):
			choice.append(ACTION_INTENDED[i]);
		
	action_max = random.choice(choice);
	return action_max;

# actual action taken (update after choice of actions )
def get_actual_action(state,action_intended):
	x = action_intended[0]
	y = action_intended[1] + ACTION_WIND_EFFECT[ state[0] ];
	return ( (x,y) );


# state after the actual action taken
# the agent could not get out of the grid world.
def get_state_next(state,action):
	state_next = ( state[0]+action[0], state[1]+action[1] );
	x = max(state_next[0],0);
	x = min(x,WIDTH-1);
	
	y = max(state_next[1],0);
	y = min(y,HEIGHT-1);
	return (x,y);


# FUNCTIONS in policy improvement

# update after every iteration and use sigma-policy 
def policy_updation(state,Q_s_a,policy):
	
	Q=[];
	for action in ACTION_INTENDED:
		Q.append(Q_s_a[state][action]);
	action_max = ACTION_INTENDED[np.argmax(Q)];
	for action in ACTION_INTENDED:
		policy[state][action] = sigma/len(ACTION_INTENDED);
	policy[state][action_max] += 1 - sigma;


# CLASS 

# CLASS Agent 

class Agent_class():
	
	def __init__(self):
		
		# q_s_a[state][action]  initial value =0;
		self.Q_s_a={};
		self.policy = copy.deepcopy( policy_target );
		
		dict_q_action={};
		for action in ACTION_INTENDED:
			dict_q_action[action] = 0;
		for x in range(0,WIDTH):
			for y in range(0,HEIGHT):
				self.Q_s_a[(x,y)] = copy.deepcopy( dict_q_action );
		
		self.action_total_set=[];
		self.state_total_set=[];
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy)
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);
		


	def finish_journey(self):
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy)
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);
				
		# state set and action set
		action_set=[];
		state_set=[];
		
		action_set.append( self.current_intended_action );
		state_set.append( self.current_state );
			
		next_state = get_state_next(self.current_state,self.current_actual_action);
		
		while next_state != STATE_GOAL:
			# get the next action;
			next_action_intended = choose_max_action(next_state,self.Q_s_a);
			next_action_actual = get_actual_action(next_state,next_action_intended)
			# policy evaluation
			self.Q_s_a[self.current_state][self.current_intended_action] = (1-alfa)*\
			self.Q_s_a[self.current_state][self.current_intended_action] + alfa* \
			(R + self.Q_s_a[next_state][next_action_intended]);
			
			#policy improvement
			policy_updation(self.current_state,self.Q_s_a,self.policy)
			# update current state and action
			self.current_intended_action = next_action_intended;
			self.current_actual_action = next_action_actual;
			self.current_state = next_state;
			# append of state set and action set
			action_set.append( self.current_intended_action );
			state_set.append( self.current_state );
			# get next state
			next_state = get_state_next(self.current_state,self.current_actual_action);
			
		# update when next state is terminal state 
		state_set.append(next_state)
		self.Q_s_a[self.current_state][self.current_intended_action] = (1-alfa)*\
		self.Q_s_a[self.current_state][self.current_intended_action] + alfa* R;
		
		self.action_total_set.append(action_set);
		self.state_total_set.append(state_set);
		
	def show_step(self):
		x=[];
		y=[];
		for state in self.state_total_set[-1]:
			x.append(state[0]);
			y.append(state[1]);
		
		plt.plot(x,y,'->')
		plt.plot(STATE_GOAL[0],STATE_GOAL[1],'*')
		plt.plot(STATE_INITIAL[0],STATE_INITIAL[1],'*')
		plt.show();
	
	def show_step_number_of_every_episode(self):
		number=[];
		for i in range(0,len(self.action_total_set)):
			number.append(len(self.action_total_set[i]));
		plt.plot(number,'-*')
		plt.xlabel('number of episodes')
		plt.ylabel('number of steps')
		plt.show();
			
		
		

# main programming 

agent = Agent_class();

for every_loop in range(0,LOOP_EPISODE):
	sigma = SIGMA*(LOOP_EPISODE-every_loop)/LOOP_EPISODE;
	#sigma = SIGMA;
	agent.finish_journey();
agent.show_step();
agent.show_step_number_of_every_episode();

Expected Sarsa 

 If we select actions in soft-greedy policy and use this policy to update Q, then it will be on-policy expected Sarsa.

If we select actions in non-greedy policy and use greedy policy to update Q, then expected Sarsa will be Q-learning.

Result

when sigma = 0.1, the influence of differnent alfa in different algorithom: 

Sarsa algorithon is not stable when alfa is large such as 0.9. But Q-learning and Expected Sarsa are stable.

Code 

##  settings

import math
import numpy as np
import random

# visualization 
import matplotlib 
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator

# list copy
import copy


# INVARIANTS 

#state   (x,y) 

STATE_INITIAL = (0,3);
STATE_GOAL = (7,3)

WIDTH = 10;
HEIGHT = 7;


# action
LEFT = (-1,0);
RIGHT = (1,0);
UP = (0,1);
DOWN = (0,-1);

ACTION_INTENDED = [LEFT,RIGHT,UP,DOWN];

# policy pi(a|s) which start at random policy, and updated after every iteration (function)
policy_target={};  # policy[(x,y)][action]

dict_actions_policy={};
for action in ACTION_INTENDED:
	dict_actions_policy[action]=1/len(ACTION_INTENDED);

for x in range(0,WIDTH):
	for y in range(0,HEIGHT):	
		policy_target[(x,y)] = copy.deepcopy(dict_actions_policy)


# reward: q_s_a(STATE_GOAL)=0, reward in every step is -1. 
R = -1; 


# special part: windy_effect, which will change the action_intended into action_actual
# 	there are different influence on different x. 
# 	1 unit influence will produce additional up in gridworld.
ACTION_WIND_EFFECT = [0,0,0,1,1,1,2,2,1,0];


# parameters we set

# sigma-policy  
SIGMA = 0.1;
# number of episode
LOOP_EPISODE = 500;
# discounting parameter
gamma = 1;
# iteration step 
ALFA =[ 0.1,0.3,0.5,0.9];


# FUNCTIONS

# FUNCTIONS in policy evaluation
def choose_action(state,policy):
	p=[];
	index=range(0,len(ACTION_INTENDED));
	for action in ACTION_INTENDED:
		p.append(policy[state][action]); 
	action_intended = ACTION_INTENDED[np.random.choice(index,p=p)];	
	return action_intended;

def choose_max_action(state,Q_s_a):
	Q=[];
	choice=[];
	for action in ACTION_INTENDED:
		Q.append(Q_s_a[state][action]);
	for i in range(0,len(Q)):
		if Q[i]==max(Q):
			choice.append(ACTION_INTENDED[i]);
		
	action_max = random.choice(choice);
	return action_max;

# actual action taken (update after choice of actions )
def get_actual_action(state,action_intended):
	x = action_intended[0]
	y = action_intended[1] + ACTION_WIND_EFFECT[ state[0] ];
	return ( (x,y) );


# state after the actual action taken
# the agent could not get out of the grid world.
def get_state_next(state,action):
	state_next = ( state[0]+action[0], state[1]+action[1] );
	x = max(state_next[0],0);
	x = min(x,WIDTH-1);
	
	y = max(state_next[1],0);
	y = min(y,HEIGHT-1);
	return (x,y);


# FUNCTIONS in policy improvement

# update after every iteration and use sigma-policy 
def policy_updation(state,Q_s_a,policy):
	
	Q=[];
	for action in ACTION_INTENDED:
		Q.append(Q_s_a[state][action]);
	actions_max = [];
	for i in range(0,len(ACTION_INTENDED)):
		if Q_s_a[state][ACTION_INTENDED[i]]==max(Q):
			actions_max.append(ACTION_INTENDED[i]);
	action_max = random.choice(actions_max);
	for action in ACTION_INTENDED:
		policy[state][action] = sigma/len(ACTION_INTENDED);
	policy[state][action_max] += 1 - sigma;

# CLASS 

# CLASS Agent 

class Agent_class():
	
	def __init__(self):
		
		# q_s_a[state][action]  initial value =0;
		self.Q_s_a={};
		self.policy = copy.deepcopy( policy_target );
		
		dict_q_action={};
		for action in ACTION_INTENDED:
			dict_q_action[action] = 0;
		for x in range(0,WIDTH):
			for y in range(0,HEIGHT):
				self.Q_s_a[(x,y)] = copy.deepcopy( dict_q_action );
		
		self.action_total_set=[];
		self.state_total_set=[];
		
		self.step_number_in_episode={};
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy)
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);


	def finish_journey(self):
		
		# current state 
		self.current_state = STATE_INITIAL;
		# current action
		self.current_intended_action = choose_action(self.current_state,self.policy);
		self.current_actual_action = get_actual_action(self.current_state,self.current_intended_action);
				
		# state set and action set
		action_set=[];
		state_set=[];
		
		action_set.append( self.current_intended_action );
		state_set.append( self.current_state );
			
		next_state = get_state_next(self.current_state,self.current_actual_action);
		
		while next_state != STATE_GOAL:
		
			# policy evaluation   average of every action
			Q_temp= (1-alfa)*self.Q_s_a[self.current_state][self.current_intended_action]\
			+alfa*R;
			for action in ACTION_INTENDED:
				Q_temp+= alfa*self.Q_s_a[next_state][action]*self.policy[next_state][action];
			self.Q_s_a[self.current_state][self.current_intended_action]=Q_temp;
			
			#policy improvement
			policy_updation(self.current_state,self.Q_s_a,self.policy)
			
			# get the next action;
			next_action_intended = choose_max_action(next_state,self.Q_s_a);
			next_action_actual = get_actual_action(next_state,next_action_intended)			
			
			# update current state and action
			self.current_intended_action = next_action_intended;
			self.current_actual_action = next_action_actual;
			self.current_state = next_state;
			# append of state set and action set
			action_set.append( self.current_intended_action );
			state_set.append( self.current_state );
			# get next state
			next_state = get_state_next(self.current_state,self.current_actual_action);
			
		# update when next state is terminal state 
		state_set.append(next_state)
		self.Q_s_a[self.current_state][self.current_intended_action] = (1-alfa)*\
		self.Q_s_a[self.current_state][self.current_intended_action] + alfa* R;
		
		self.action_total_set.append(action_set);
		self.state_total_set.append(state_set);
		
		
	def show_step(self):
		x=[];
		y=[];
		for state in self.state_total_set[-1]:
			x.append(state[0]);
			y.append(state[1]);
		
		plt.plot(x,y,'->')
		plt.plot(STATE_GOAL[0],STATE_GOAL[1],'*')
		plt.plot(STATE_INITIAL[0],STATE_INITIAL[1],'*')
		plt.show();
	
	def step_number_of_every_episode(self):
		number=[];
		for i in range(0,len(self.action_total_set)):
			number.append(len(self.action_total_set[i]));
		self.step_number_in_episode=number;
		
	def show_step_number_of_every_episode(self):
		number=[];
		for i in range(0,len(self.action_total_set)):
			number.append(len(self.action_total_set[i]));	
		plt.plot(number,'-*')
		plt.xlabel('number of episodes')
		plt.ylabel('number of steps')
		plt.show();
			

# main programming 
number_set =[];
for alfa in ALFA:
	
	agent = Agent_class();

	for every_loop in range(0,LOOP_EPISODE):
		#sigma = SIGMA*(LOOP_EPISODE-every_loop)/LOOP_EPISODE;
		sigma = SIGMA;
		agent.finish_journey();
	#agent.show_step();
	agent.step_number_of_every_episode();
	number_set.append(agent.step_number_in_episode);
for i in range(0,len(ALFA)):
	plt.plot(number_set[i],label='expected Sarsa '+'alfa='+str(ALFA[i]))
plt.legend();
plt.show();

最適当承诺
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
深度强化学习之Policy Gradient & Actor-Critic Model & A3C
I AM BACK
01-12 7214
policy gradient actor-critic A3Cpolicy gradient在之前的DQN是进行value function的approximation，是value-vased RL。这次要讲的是policy-based RL。对于RL的policy直接进行参数表示：比如可以输入state（一幅图片，一个feature vector），然后由带有参数的某种算法（比如linear
[RL] Expected Sarsa Algorithm (in Python)
weixin_47553356的博客
03-07 401
Numpy imported Expected Sarsa在Grid问题中的应用 class ExpectedSarsaAgent(BaseAgent): def agent_init(self, agent_init_info): """Setup for the agent called when the experiment first starts. Args: agent_init_info (dict), the pa.
Q-learning、Expected Sarsa、Double Learning
WZX_Hello的博客
04-21 505
Contents一级目录二级目录三级目录 一级目录 二级目录 三级目录
强化学习之Monte Carol
RosebudTT的博客
05-19 487
这篇文章就用MC方法来玩21点扑克牌游戏。 规则简介 在这里智能体(Agent)扮演玩家(Player)，对方是庄家(Dealer)。 点数(Score)：2-10的点数为牌面数字；J，Q，K是10点；A有两种算法，1或者11，算11总点数不超过21时则必须算成11(usable)，否则算作1。 庄家需要亮(Show)一张牌，玩家根据自己手中的牌和庄家亮的牌决定是要牌(hits)还是停牌(sticks)。 庄家要牌和停牌的规则是固定的，即点数小于17必须要牌，否则停牌。 爆牌(goes bust
动手学强化学习（四）：时序差分算法 （Temporal Difference）
与君共勉，一起学习
05-06 5565
强化学习基础篇（四）：时序差分算法 （Temporal Difference）1. 简介2. 时序差分方法3. Sarsa 算法4. 多步 Sarsa 算法5. Q-learning 算法在线策略算法与离线策略算法6. 总结 文章转于 伯禹学习平台-动手学强化学习 （强推） 与君共勉，一起学习。 1. 简介   动态规划算法要求马尔可夫决策过程是已知的，即要求与智能体交互的环境是完全已知的（例如迷宫或者给定规则的网格世界）。在此条件下，智能体其实并不需要和环境真正交互来采样数据，直接用动态规划算法就可以解
Temporal Uncertainty of Geographical Information
01-19
Temporal Uncertainty of Geographical Information，舒红，，Temporal uncertainty is a crossing point of temporal and error-aware geographical information systems. In Geoinformatics, temporal uncertainty...
Spatio-temporal analysis of type 2 diabetes mellitus based on differential expression networks
02-09
Here we perform a spatio-temporal analysis on T2DM by developing a new form of molecular network, i.e. 'differential expression network' (DEN), which can reflect phenotype differences at ...
论文研究-Semantics of Framed Temporal Logic Programs.pdf
08-27
框架时序逻辑程序语义，段振华，杨潇潇，本文研究框架时序逻辑程序的语义。为此介绍了投影时序逻辑和该逻辑下的可执行子集，在该语言的基础上，引入了框架技术。非框架程
Study and application of temporal index technology
02-20
This paper addresses the mathematical relation on a set of periods and temporal indexing construc- tions as well as their applications. First we introduce two concepts, i.e. the temporal connection ...
论文研究-Satisfiablity of Propositional Projection Temporal Logic.pdf
08-16
命题投影时序逻辑的可满足性，段振华，张丽，本文研究命题投影时序逻辑的可满足性的判定过程. 文中给出了该逻辑的语法,语义及该逻辑公式的标准范式.研究了该逻辑公式在有穷模��
强化学习（RLAI）读书笔记第六章差分学习（TD-learning）
无所知的博客
09-04 5365
第六章：Temporal-Difference Learning TD-learning算法是强化学习中一个独具特色而又核心的想法，结合了蒙特卡洛算法和动态规划的想法。和MC一样不需要环境模型直接从sample里学习，也像DP一样使用bootstrap通过别的状态值的估计更新当前状态值。首先也关注的是给定策略进行prediction或者叫policy evaluation。对于control（找...
强化学习（RLAI）读书笔记第七章n步自举（n-step Bootstrapping）
无所知的博客
09-05 3118
第七章：n-step Bootstrapping 这一章中我们把蒙特卡洛算法(MC)和一步差分算法(one-step TD)统一起来。MC算法和一步TD算法都不可能永远是最好的。这两个方法都是比较极端的形式，可能使用它们中间形式会更好一些。另一个看待n步TD算法的优势的角度是它解决了固定时间步骤的缺点。比如一步TD算法固定了每次选择动作和更新值的时间间隔。很多应用中想要把发生的改变快速更新到值函...
强化学习系列（六）：时间差分算法（Temporal-Difference Learning)
热门推荐
LagrangeSK的博客
07-28 1万+
一、前言 在第五章中，我们提到了求解环境模型未知MDP的方法——Monte Carlo，但该方法是每个episode 更新一次（episode-by-episode)。本章介绍一种单步更新的求解环境模型未知MDP的方法——Temporal-Difference Learning(TD)。TD（0）算法结合了DP 和Monte Carlo算法的优点，不仅可以单步更新，而且可以根据经验进行学习。 ...
强化学习系列12：使用julia训练深度强化模型
kittyzc的博客
06-16 848
1. 介绍 使用flux作为深度学习的框架，入门代码： using ReinforcementLearning run(E`JuliaRL_BasicDQN_CartPole`) 显示： This experiment uses three dense layers to approximate the Q value. The testing environment is CartPoleEnv. Agent and statistic info will be saved to: `/ho
lamda 对比两个list_解剖[强化学习]Double Learning本质并对比Q-Learning与期望Sarsa
weixin_29466339的博客
12-20 175
引言：本文首先通过引出Q-learning的最大化偏差问题，然后从solve最大化偏差的角度出发分别阐述Expected Sarsa和Double Q-learning，并从实验中Q表的更新过程来深入剖析Double Learning思想的本质。最后实验对比了Double Learning、Q Learning、Expected Sarsa以及Action Distribution在案例中的表现....
强化学习经典算法笔记(二)：策略迭代算法Policy Iteration
hhy_csdn的博客
04-08 8248
强化学习经典算法笔记——策略迭代算法   上一篇讲了价值迭代算法，这一篇介绍另一个动态规划算法——策略迭代算法（Policy Iteration）。 简单介绍   Value Iteration的思路是：先迭代找出一个最优的Value Function，然后再根据Value Function迭代出一个最优策略。   Policy Iteration的思路是反着的，首先给定一个初始化的策略函数，一般...
On-policy Sarsa算法与Off-policy Q learning对比
庞琳卓的博客
05-19 1万+
强化学习中Sarsa算法（on policy）与Q learning（off policy）的详细对比。
强化学习(reinforcement learning)学习笔记(二)——值迭代与策略迭代
trillion_power的博客
12-29 1万+
上一次我分享了强化学习相关基础概念，推导了Bellman方程。这次我们来学习求解强化学习模型的2种思路——值迭代与策略迭代动态规划这里面我要简单介绍一下动态规划，因为严格来说，值迭代与策略迭代是用来解决动态规划问题的两种规划方法。而强化学习又有另外一个昵称——就是拟动态规划。说白了强化学习就是模拟动态规划算法。用一句话来总结动态规划就是，对一个复杂问题给出一个一般性的解决办法。它主要由两个性质: 最
temporal kit IndexError: list index out of range
最新发布
08-24
根据提供的引用内容，temporal kit IndexError: list index out of range 是一个错误信息，提示列表索引超出范围。根据引用内容，无法确定具体的上下文和代码逻辑。为了更好地帮助解决该错误，您可以提供更多关于...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值