Trap maze:
MDP is used to solve the problem of trap maze. In the maze, in addition to the starting point and the end point, there is also the tendency of trap. When the trap area or the outer wall is touched, it will directly end and cannot continue. Moreover, walking in the maze needs to consume the ability, so we hope to use MDP algorithm to find the optimal shortest path.
In addition, mazes can interfere with your nerves, and when you decide to take action in one direction, there’s a small chance you’ll go the wrong way.
MAZE:
[[-100,-100, -100, -100,-100, -100 ],
[-100,-1, -1, -1, -1, -100 ],
[-100,-1, -100, -100, -100, -100 ],
[-100,-1, -100, -1, -1, -100 ],
[-100,-1, -1, -1, 1, -100 ],
[-100,-100, -100, -100,-100, -100]]
Set walls and traps with R of -100, end point with R of 1, and consume 1 energy without taking a step.
Goal:
Find the optimal path to the destination
[[0. 1. 1. 1. 1. 0.]
[3. 1. 2. 2. 2. 2.]
[3. 1. 2. 1. 1. 0.]
[3. 1. 1. 1. 1. 2.]
[3. 3. 3. 3. 1. 2.]
[0. 0. 0. 0. 0. 0.]]
0: go up
1: go down
2: go left
3: go right
See the code
Limits :
- MDP algorithm for finite space maze can be easily solved, but it is difficult to solve infinite space maze. (There’s an infinite number of states)
- For dynamic mazes, or multi-task mazes (which must go through a path), MDP algorithm is not easy to implement.
代码实现
import numpy as np
class maze_mdp:
def __init__(self,maze,gamma):
self.maze = maze
self.row = maze.shape[0]
self.col = maze.shape[1]
self.r_s = np.array([x for y in maze for x in y])
self.end = np.where(self.r_s > 0)[0][0]
self.trap = np.where(self.r_s < -1)[0]
self.v_s = np.zeros(len(self.r_s))
self.maze_number = np.arange(0, len(self.r_s)).reshape(maze.shape)
# 0 : up 85%
# 1 : down 85%
# 2 : left 85%
# 3 : right 85%
self.p1 = 0.85
self.p2 = 0.05
self.action = np.zeros(len(self.r_s))
self.gamma = gamma
def go_up(self,x):
if 0 <= x-self.col:
return x-self.col
else:
return x
def go_left(self,x):
if x in self.maze_number[:,0]:
return x
else:
return x-1
def go_right(self,x):
if x in self.maze_number[:,-1]:
return x
else:
return x+1
def go_down(self,x):
if x+self.col <= self.end:
return x+self.col
else:
return x
def go(self,x):
return self.go_up(x),self.go_down(x),self.go_left(x),self.go_right(x)
def update_value_pai(self):
old_v_s = self.v_s.copy()
for i in range(len(self.r_s)):
if i == self.end or i in self.trap:
self.v_s[i] = 0
continue
up , down , left ,right = self.go(i)
if self.action[i] == 0:
self.v_s[i] = self.p1 *(self.r_s[up] + self.gamma *old_v_s[up]) + \
self.p2 *(self.r_s[down] + self.gamma *old_v_s[down]) + \
self.p2 *(self.r_s[left] + self.gamma *old_v_s[left]) + \
self.p2 *(self.r_s[right] + self.gamma *old_v_s[right])