CS229 吴恩达机器学习 习题大作业答案 problem sets 04 PS04（第6题，欢迎指教）强化学习 Reinforcement Learning MDP 马尔可夫决策

最新推荐文章于 2024-07-28 00:44:53 发布
最新推荐文章于 2024-07-28 00:44:53 发布
阅读量1.2k 收藏 1
点赞数 1
分类专栏： CS229 文章标签： 机器学习 人工智能 python 深度学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_26928055/article/details/124538983
版权

6. Reinforcement Learning: The inverted pendulum

首先写出simulator（作业中已提供，直接复制过来）：

import matplotlib.pyplot as plt
import matplotlib.patches as patches
from math import sin, cos, pi

class CartPole:
    def __init__(self, physics):
        self.physics = physics
        self.mass_cart = 1.0
        self.mass_pole = 0.3
        self.mass = self.mass_cart + self.mass_pole
        self.length = 0.7 # actually half the pole length 也就是质心位置
        self.pole_mass_length = self.mass_pole * self.length # 质量矩，乘以重力因子就是重力矩（重力的力矩）

    def simulate(self, action, state_tuple):
        """
        Simulation dynamics of the cart-pole system

        Parameters
        ----------
        action : int
            Action represented as 0 or 1
        state_tuple : tuple
            Continuous vector of x, x_dot, theta, theta_dot

        Returns
        -------
        new_state : tuple
            Updated state vector of new_x, new_x_dot, nwe_theta, new_theta_dot
        """
        x, x_dot, theta, theta_dot = state_tuple
        costheta, sintheta = cos(theta), sin(theta)
        # costheta, sintheta = cos(theta * 180 / pi), sin(theta * 180 / pi)

        # calculate force based on action
        force = self.physics.force_mag if action > 0 else (-1 * self.physics.force_mag)

        # intermediate calculation
        temp = (force + self.pole_mass_length * theta_dot * theta_dot * sintheta) / self.mass
        theta_acc = (self.physics.gravity * sintheta - temp * costheta) / (self.length * (4/3 - self.mass_pole * costheta * costheta / self.mass))

        x_acc = temp - self.pole_mass_length * theta_acc * costheta / self.mass

        # return new state variable using Euler's method
        new_x = x + self.physics.tau * x_dot
        new_x_dot = x_dot + self.physics.tau * x_acc
        new_theta = theta + self.physics.tau * theta_dot
        new_theta_dot = theta_dot + self.physics.tau * theta_acc
        new_state = (new_x, new_x_dot, new_theta, new_theta_dot)

        return new_state

    def get_state(self, state_tuple):
        """
        Discretizes the continuous state vector. The current discretization
        divides x into 3, x_dot into 3, theta into 6 and theta_dot into 3
        categories. A finer discretization produces a larger state space
        but allows for a better policy

        Parameters
        ----------
        state_tuple : tuple
            Continuous vector of x, x_dot, theta, theta_dot

        Returns
        -------
        state : int
            Discretized state value
        """
        x, x_dot, theta, theta_dot = state_tuple
        # parameters for state discretization in get_state
        # convert degrees to radians
        one_deg = pi / 180
        six_deg = 6 * pi / 180
        twelve_deg = 12 * pi / 180
        fifty_deg = 50 * pi / 180

        total_states = 163
        state = 0

        if x < -2.4 or x > 2.4 or theta < -twelve_deg or theta > twelve_deg:
            state = total_states - 1 # to signal failure
        else:
            # x: 3 categories
            if x < -1.5:
                state = 0
            elif x < 1.5:
                state = 1
            else:
                state = 2
            # x_dot: 3 categories
            if x_dot < -0.5:
                pass
            elif x_dot < 0.5:
                state += 3
            else:
                state += 6
            # theta: 6 categories
            if theta < -six_deg:
                pass
            elif theta < -one_deg:
                state += 9
            elif theta < 0:
                state += 18
            elif theta < one_deg:
                state += 27
            elif theta < six_deg:
                state += 36
            else:
                state += 45
            # theta_dot: 3 categories
            if theta_dot < -fifty_deg:
                pass
            elif theta_dot < fifty_deg:
                state += 54
            else:
                state += 108
        # state += 1 # converting from MATLAB 1-indexing to 0-indexing
        return state

    def show_cart(self, state_tuple, pause_time):
        """
        Given the `state_tuple`, displays the cart-pole system.

        Parameters
        ----------
        state_tuple : tuple
            Continuous vector of x, x_dot, theta, theta_dot
        pause_time : float
            Time delay in seconds

        Returns
        -------
        """
        x, x_dot, theta, theta_dot = state_tuple
        X = [x, x + 4*self.length * sin(theta)]
        Y = [0, 4*self.length * cos(theta)]
        plt.close('all')
        fig, ax = plt.subplots(1)
        plt.ion()
        ax.set_xlim(-3, 3)
        ax.set_ylim(-0.5, 3.5)
        ax.plot(X, Y)
        cart = patches.Rectangle((x - 0.4, -0.25), 0.8, 0.25,
                        linewidth=1, edgecolor='k', facecolor='cyan')
        base = patches.Rectangle((x - 0.01, -0.5), 0.02, 0.25,
                        linewidth=1, edgecolor='k', facecolor='r')
        ax.add_patch(cart)
        ax.add_patch(base)
        x_dot_str, theta_str, theta_dot_str = '\\dot{x}', '\\theta', '\\dot{\\theta}'
        ax.set_title('x: %.3f, $%s$: %.3f, $%s$: %.3f, $%s$: %.3f'\
                                %(x, x_dot_str, x_dot, theta_str, theta, theta_dot_str, x))
        plt.show()
        plt.pause(pause_time)

class Physics:
    gravity = 9.8
    force_mag = 10.0
    tau = 0.02 # seconds between state updates

接下来是MDP强化学习过程：

# from env import CartPole, Physics
import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import lfilter

"""
Parts of the code (cart and pole dynamics, and the state
discretization) are inspired from code available at the RL repository
http://www-anw.cs.umass.edu/rlr/domains.html

Briefly, the cart-pole system is described in `p06_cartpole.py`. The main
simulation loop in this file calls the `simulate()` function for
simulating the pole dynamics, `get_state()` for discretizing the
otherwise continuous state space in discrete states, and `show_cart()`
for display.

Some useful parameters are listed below:

`NUM_STATES`: Number of states in the discretized state space
You must assume that states are numbered 0 through `NUM_STATES` - 1. The
state numbered `NUM_STATES` - 1 (the last one) is a special state that
marks the state when the pole has been judged to have fallen (or when
the cart is out of bounds). However, you should NOT treat this state
any differently in your code. Any distinctions you need to make between
states should come automatically from your learning algorithm.

After each simulation cycle, you are supposed to update the transition
counts and rewards observed. However, you should not change either
your value function or the transition probability matrix at each
cycle.

Whenever the pole falls, a section of your code below will be
executed. At this point, you must use the transition counts and reward
observations that you have gathered to generate a new model for the MDP
(i.e. transition probabilities and state rewards). After that, you
must use value iteration to get the optimal value function for this MDP
model.

`TOLERANCE`: Controls the convergence criteria for each value iteration
run. In value iteration, you can assume convergence when the maximum
absolute change in the value function at any state in an iteration
becomes lower than `TOLERANCE.

You need to write code that chooses the best action according
to your current value function, and the current model of the MDP. The
action must be either 0 or 1 (corresponding to possible directions of
pushing the cart)

Finally, we assume that the simulation has converged when
`NO_LEARNING_THRESHOLD` consecutive value function computations all
converged within one value function iteration. Intuitively, it seems
like there will be little learning after this, so we end the simulation
here, and say the overall algorithm has converged.


Learning curves can be generated by calling a code snippet at the end
(it assumes that the learning was just executed, and the array
`time_steps_to_failure` that records the time for which the pole was
balanced before each failure is in memory). `num_failures` is a variable
that stores the number of failures (pole drops / cart out of bounds)
till now.

Other parameters in the code are described below:

`GAMMA`: Discount factor to be used

The following parameters control the simulation display; you dont
really need to know about them:

`pause_time`: Controls the pause between successive frames of the
display. Higher values make your simulation slower.
`min_trial_length_to_start_display`: Allows you to start the display only
after the pole has been successfully balanced for at least this many
trials. Setting this to zero starts the display immediately. Choosing a
reasonably high value (around 100) can allow you to rush through the
initial learning quickly, and start the display only after the
performance is reasonable.
"""


def initialize_mdp_data(num_states):
    """
    Return a variable that contains all the parameters/state you need for your MDP.
    Feel free to use whatever data type is most convenient for you (custom classes, tuples, dicts, etc)

    Assume that no transitions or rewards have been observed.
    Initialize the value function array to small random values (0 to 0.10, say).
    Initialize the transition probabilities uniformly (ie, probability of
        transitioning for state x to state y using action a is exactly
        1/num_states).
    Initialize all state rewards to zero.

    Args:
        num_states: The number of states

    Returns: The initial MDP parameters
    """
    transition_counts = np.zeros((num_states, num_states, 2))
    transition_probs = np.ones((num_states, num_states, 2)) / num_states
    # Index zero is count of rewards being -1 , index 1 is count of total num state is reached
    reward_counts = np.zeros((num_states, 2))
    reward = np.zeros(num_states)
    value = np.random.rand(num_states) * 0.1

    return {
        'transition_counts': transition_counts,
        'transition_probs': transition_probs,
        'reward_counts': reward_counts,
        'reward': reward,
        'value': value,
        'num_states': num_states,
    }


def choose_action(state, mdp_data):
    """
    Choose the next action (0 or 1) that is optimal according to your current
    mdp_data. When there is no optimal action, return a random action.

    Args:
        state: The current state in the MDP
        mdp_data: The parameters for your MDP. See initialize_mdp_data.

    Returns:
        0 or 1 that is optimal according to your current MDP
    """

    # *** START CODE HERE ***
    result = mdp_data['transition_probs'][state] * mdp_data['value'].reshape(-1, 1)
    result = result.sum(axis=0)
    if result[0] == result[1]:
        return np.random.randint(2)
    else:
        return result.argmax()
    # *** END CODE HERE ***


def update_mdp_transition_counts_reward_counts(mdp_data, state, action, new_state, reward):
    """
    Update the transition count and reward count information in your mdp_data.
    Do not change the other MDP parameters (those get changed later).

    Record the number of times `state, action, new_state` occurs.
    Record the rewards for every `new_state`
    (since rewards are -1 or 0, you just need to record number of times reward -1 is seen in 'reward_counts' index new_state,0)
    Record the number of time `new_state` was reached (in 'reward_counts' index new_state,1)

    Args:
        mdp_data: The parameters of your MDP. See initialize_mdp_data.
        state: The state that was observed at the start.
        action: The action you performed.
        new_state: The state after your action.
        reward: The reward after your action (i.e. reward corresponding to new_state).

    Returns:
        Nothing
    """

    # *** START CODE HERE ***
    mdp_data['transition_counts'][state, new_state, action] += 1
    if reward == -1:
        mdp_data['reward_counts'][new_state] += 1
    else:
        mdp_data['reward_counts'][new_state, 1] += 1
    # *** END CODE HERE ***

    # This function does not return anything
    return


def update_mdp_transition_probs_reward(mdp_data):
    """
    Update the estimated transition probabilities and reward values in your MDP.

    Make sure you account for the case when a state-action pair has never
    been tried before, or the state has never been visited before. In that
    case, you must not change that component (and thus keep it at the
    initialized uniform distribution).
    
    Args:
        mdp_data: The data for your MDP. See initialize_mdp_data.

    Returns:
        Nothing

    """

    # *** START CODE HERE ***
    num_states = mdp_data['transition_counts'].shape[0]
    num_actions = mdp_data['transition_counts'].shape[2]

    num_state_action_pairs = mdp_data['transition_counts'].sum(axis=1)  # (states, action)
    for origin_state in range(num_states):
        for action in range(num_actions):
            if num_state_action_pairs[origin_state, action] != 0:
                mdp_data['transition_probs'][origin_state, :, action] = mdp_data['transition_counts'][origin_state, :,
                                                                        action] / num_state_action_pairs[
                                                                            origin_state, action]

    for state in range(num_states):
        num_recorded_states = mdp_data['reward_counts'][state, 1]
        if num_recorded_states != 0:
            mdp_data['reward'][state] = -1 * mdp_data['reward_counts'][state, 0] / num_recorded_states
    # *** END CODE HERE ***

    # This function does not return anything
    return


def update_mdp_value(mdp_data, tolerance, gamma):
    """
    Update the estimated values in your MDP.

    Perform value iteration using the new estimated model for the MDP.
    The convergence criterion should be based on `TOLERANCE` as described
    at the top of the file.

    Return true if it converges within one iteration.
    
    Args:
        mdp_data: The data for your MDP. See initialize_mdp_data.
        tolerance: The tolerance to use for the convergence criterion.
        gamma: Your discount factor.

    Returns:
        True if the value iteration converged in one iteration

    """

    # *** START CODE HERE ***
    error = np.inf
    it = 0
    reward = mdp_data['reward']
    transition_probs = mdp_data['transition_probs']

    while error >= tolerance:
        old_value = mdp_data['value']
        # 注意numpy.dot函数，当第二个数组的维度大于等于2时，其实是将前面数组的末尾轴与后面数组的倒数第二个轴做乘法求和，如何对于矩阵乘法才“恰好”
        # 此处的更新方法是讲义中所谓的“synchronous”(另一种叫做asynchronous),即“同步”更新方式
        new_value = reward + gamma * old_value.dot(transition_probs).max(axis=1)
        # 等价：value = new_value
        mdp_data['value'] = new_value
        error = np.max(np.abs(new_value - old_value))
        it += 1

    return it == 1

    # *** END CODE HERE ***


def main(plot=True, seed=0):
    # Seed the randomness of the simulation so this outputs the same thing each time
    seed = seed
    np.random.seed(seed)

    # Simulation parameters
    pause_time = 0.0001
    min_trial_length_to_start_display = 100
    display_started = min_trial_length_to_start_display == 0

    NUM_STATES = 163
    GAMMA = 0.995
    TOLERANCE = 0.01
    NO_LEARNING_THRESHOLD = 20

    # Time cycle of the simulation
    time = 0

    # These variables perform bookkeeping (how many cycles was the pole
    # balanced for before it fell). Useful for plotting learning curves.
    time_steps_to_failure = []
    num_failures = 0
    time_at_start_of_current_trial = 0

    # You should reach convergence well before this
    max_failures = 500

    # Initialize a cart pole
    cart_pole = CartPole(Physics())

    # Starting `state_tuple` is (0, 0, 0, 0)
    # x, x_dot, theta, theta_dot represents the actual continuous state vector
    x, x_dot, theta, theta_dot = 0.0, 0.0, 0.0, 0.0
    state_tuple = (x, x_dot, theta, theta_dot)

    # `state` is the number given to this state, you only need to consider
    # this representation of the state
    state = cart_pole.get_state(state_tuple)
    if min_trial_length_to_start_display == 0 or display_started == 1:
        cart_pole.show_cart(state_tuple, pause_time)

    mdp_data = initialize_mdp_data(NUM_STATES)

    # This is the criterion to end the simulation.
    # You should change it to terminate when the previous
    # 'NO_LEARNING_THRESHOLD' consecutive value function computations all
    # converged within one value function iteration. Intuitively, it seems
    # like there will be little learning after this, so end the simulation
    # here, and say the overall algorithm has converged.

    consecutive_no_learning_trials = 0
    while consecutive_no_learning_trials < NO_LEARNING_THRESHOLD:

        action = choose_action(state, mdp_data)

        # Get the next state by simulating the dynamics
        state_tuple = cart_pole.simulate(action, state_tuple)
        # x, x_dot, theta, theta_dot = state_tuple

        # Increment simulation time
        time = time + 1

        # Get the state number corresponding to new state vector
        new_state = cart_pole.get_state(state_tuple)
#         if display_started == 1:
#             cart_pole.show_cart(state_tuple, pause_time)

        # reward function to use - do not change this!
        if new_state == NUM_STATES - 1:
            R = -1
        else:
            R = 0

        # 不产生“失败”情况时不需要进行MDP模型的更新，只需要记录相应的计数即可，等到遇到了“失败”情况，这些非失败情况的计数会被考虑进去
        update_mdp_transition_counts_reward_counts(mdp_data, state, action, new_state, R)

        # Recompute MDP model whenever pole falls
        # Compute the value function V for the new model
        if new_state == NUM_STATES - 1:

            update_mdp_transition_probs_reward(mdp_data)

            converged_in_one_iteration = update_mdp_value(mdp_data, TOLERANCE, GAMMA)

            if converged_in_one_iteration:
                consecutive_no_learning_trials = consecutive_no_learning_trials + 1
            else:
                consecutive_no_learning_trials = 0

        # Do NOT change this code: Controls the simulation, and handles the case
        # when the pole fell and the state must be reinitialized.
        if new_state == NUM_STATES - 1:
            num_failures += 1
            if num_failures >= max_failures:
                break
            print('[INFO] Failure number {}'.format(num_failures))
            time_steps_to_failure.append(time - time_at_start_of_current_trial)
            # time_steps_to_failure[num_failures] = time - time_at_start_of_current_trial
            time_at_start_of_current_trial = time

            if time_steps_to_failure[num_failures - 1] > min_trial_length_to_start_display:
                display_started = 1

            # Reinitialize state
            # x = 0.0
            x = -1.1 + np.random.uniform() * 2.2
            x_dot, theta, theta_dot = 0.0, 0.0, 0.0
            state_tuple = (x, x_dot, theta, theta_dot)
            state = cart_pole.get_state(state_tuple)
        else:
            state = new_state

    if plot:
        # plot the learning curve (time balanced vs. trial)
        log_tstf = np.log(np.array(time_steps_to_failure))
        plt.plot(np.arange(len(time_steps_to_failure)), log_tstf, 'k')
        window = 30
        w = np.array([1 / window for _ in range(window)])
        weights = lfilter(w, 1, log_tstf)
        x = np.arange(window // 2, len(log_tstf) - window // 2)
        plt.plot(x, weights[window:len(log_tstf)], 'r--')
        plt.xlabel('Num failures')
        plt.ylabel('Log of num steps to failure')
        plt.title('seed = {}'.format(seed))
        plt.savefig('output/control_{}.png'.format(seed))

    return np.array(time_steps_to_failure)

main()

[INFO] Failure number 1
[INFO] Failure number 2
[INFO] Failure number 3
[INFO] Failure number 4
[INFO] Failure number 5
[INFO] Failure number 6
[INFO] Failure number 7
[INFO] Failure number 8
[INFO] Failure number 9
[INFO] Failure number 10
[INFO] Failure number 11
[INFO] Failure number 12
[INFO] Failure number 13
[INFO] Failure number 14
[INFO] Failure number 15
[INFO] Failure number 16
[INFO] Failure number 17
[INFO] Failure number 18
[INFO] Failure number 19
[INFO] Failure number 20
[INFO] Failure number 21
[INFO] Failure number 22
[INFO] Failure number 23
[INFO] Failure number 24
[INFO] Failure number 25
[INFO] Failure number 26
[INFO] Failure number 27
[INFO] Failure number 28
[INFO] Failure number 29
[INFO] Failure number 30
[INFO] Failure number 31
[INFO] Failure number 32
[INFO] Failure number 33
[INFO] Failure number 34
[INFO] Failure number 35
[INFO] Failure number 36
[INFO] Failure number 37
[INFO] Failure number 38
[INFO] Failure number 39
[INFO] Failure number 40
[INFO] Failure number 41
[INFO] Failure number 42
[INFO] Failure number 43
[INFO] Failure number 44
[INFO] Failure number 45
[INFO] Failure number 46
[INFO] Failure number 47
[INFO] Failure number 48
[INFO] Failure number 49
[INFO] Failure number 50
[INFO] Failure number 51
[INFO] Failure number 52
[INFO] Failure number 53
[INFO] Failure number 54
[INFO] Failure number 55
[INFO] Failure number 56
[INFO] Failure number 57
[INFO] Failure number 58
[INFO] Failure number 59
[INFO] Failure number 60
[INFO] Failure number 61
[INFO] Failure number 62
[INFO] Failure number 63
[INFO] Failure number 64
[INFO] Failure number 65
[INFO] Failure number 66
[INFO] Failure number 67
[INFO] Failure number 68
[INFO] Failure number 69
[INFO] Failure number 70
[INFO] Failure number 71
[INFO] Failure number 72
[INFO] Failure number 73
[INFO] Failure number 74
[INFO] Failure number 75
[INFO] Failure number 76
[INFO] Failure number 77
[INFO] Failure number 78
[INFO] Failure number 79
[INFO] Failure number 80
[INFO] Failure number 81
[INFO] Failure number 82
[INFO] Failure number 83
[INFO] Failure number 84
[INFO] Failure number 85
[INFO] Failure number 86
[INFO] Failure number 87
[INFO] Failure number 88
[INFO] Failure number 89
[INFO] Failure number 90
[INFO] Failure number 91
[INFO] Failure number 92
[INFO] Failure number 93
[INFO] Failure number 94
[INFO] Failure number 95
[INFO] Failure number 96
[INFO] Failure number 97
[INFO] Failure number 98
[INFO] Failure number 99
[INFO] Failure number 100
[INFO] Failure number 101
[INFO] Failure number 102
[INFO] Failure number 103
[INFO] Failure number 104
[INFO] Failure number 105
[INFO] Failure number 106
[INFO] Failure number 107
[INFO] Failure number 108
[INFO] Failure number 109
[INFO] Failure number 110
[INFO] Failure number 111
[INFO] Failure number 112
[INFO] Failure number 113
[INFO] Failure number 114
[INFO] Failure number 115
[INFO] Failure number 116
[INFO] Failure number 117
[INFO] Failure number 118
[INFO] Failure number 119
[INFO] Failure number 120
[INFO] Failure number 121
[INFO] Failure number 122
[INFO] Failure number 123
[INFO] Failure number 124
[INFO] Failure number 125
[INFO] Failure number 126
[INFO] Failure number 127
[INFO] Failure number 128
[INFO] Failure number 129
[INFO] Failure number 130
[INFO] Failure number 131
[INFO] Failure number 132
[INFO] Failure number 133
[INFO] Failure number 134
[INFO] Failure number 135
[INFO] Failure number 136
[INFO] Failure number 137
[INFO] Failure number 138
[INFO] Failure number 139
[INFO] Failure number 140
[INFO] Failure number 141
[INFO] Failure number 142
[INFO] Failure number 143
[INFO] Failure number 144
[INFO] Failure number 145
[INFO] Failure number 146
[INFO] Failure number 147
[INFO] Failure number 148
[INFO] Failure number 149
[INFO] Failure number 150
[INFO] Failure number 151
[INFO] Failure number 152
[INFO] Failure number 153
[INFO] Failure number 154
[INFO] Failure number 155
[INFO] Failure number 156
[INFO] Failure number 157
[INFO] Failure number 158
[INFO] Failure number 159
[INFO] Failure number 160
[INFO] Failure number 161
[INFO] Failure number 162
[INFO] Failure number 163
[INFO] Failure number 164
[INFO] Failure number 165
[INFO] Failure number 166
[INFO] Failure number 167
[INFO] Failure number 168
[INFO] Failure number 169
[INFO] Failure number 170
[INFO] Failure number 171
[INFO] Failure number 172
[INFO] Failure number 173
[INFO] Failure number 174
[INFO] Failure number 175
[INFO] Failure number 176
[INFO] Failure number 177
[INFO] Failure number 178
[INFO] Failure number 179
[INFO] Failure number 180
[INFO] Failure number 181
[INFO] Failure number 182
[INFO] Failure number 183
[INFO] Failure number 184
[INFO] Failure number 185
[INFO] Failure number 186
[INFO] Failure number 187
[INFO] Failure number 188
[INFO] Failure number 189
[INFO] Failure number 190
[INFO] Failure number 191
[INFO] Failure number 192
[INFO] Failure number 193
[INFO] Failure number 194
[INFO] Failure number 195
[INFO] Failure number 196
[INFO] Failure number 197
[INFO] Failure number 198
[INFO] Failure number 199
[INFO] Failure number 200
[INFO] Failure number 201
[INFO] Failure number 202
[INFO] Failure number 203
[INFO] Failure number 204
[INFO] Failure number 205
[INFO] Failure number 206
[INFO] Failure number 207
[INFO] Failure number 208
[INFO] Failure number 209
[INFO] Failure number 210
[INFO] Failure number 211
[INFO] Failure number 212
[INFO] Failure number 213
[INFO] Failure number 214
[INFO] Failure number 215
[INFO] Failure number 216
[INFO] Failure number 217
[INFO] Failure number 218
[INFO] Failure number 219
[INFO] Failure number 220
[INFO] Failure number 221
[INFO] Failure number 222
[INFO] Failure number 223
[INFO] Failure number 224
[INFO] Failure number 225
[INFO] Failure number 226
[INFO] Failure number 227
[INFO] Failure number 228
[INFO] Failure number 229
[INFO] Failure number 230
[INFO] Failure number 231
[INFO] Failure number 232
[INFO] Failure number 233
[INFO] Failure number 234
[INFO] Failure number 235
[INFO] Failure number 236
[INFO] Failure number 237
[INFO] Failure number 238
[INFO] Failure number 239
[INFO] Failure number 240
[INFO] Failure number 241
[INFO] Failure number 242
[INFO] Failure number 243
[INFO] Failure number 244
[INFO] Failure number 245
[INFO] Failure number 246
[INFO] Failure number 247
[INFO] Failure number 248
[INFO] Failure number 249
[INFO] Failure number 250
[INFO] Failure number 251
[INFO] Failure number 252
[INFO] Failure number 253
[INFO] Failure number 254
[INFO] Failure number 255
[INFO] Failure number 256
[INFO] Failure number 257
[INFO] Failure number 258
[INFO] Failure number 259
[INFO] Failure number 260
[INFO] Failure number 261
[INFO] Failure number 262
[INFO] Failure number 263
[INFO] Failure number 264
[INFO] Failure number 265
[INFO] Failure number 266
[INFO] Failure number 267
[INFO] Failure number 268
[INFO] Failure number 269
[INFO] Failure number 270
[INFO] Failure number 271
[INFO] Failure number 272
[INFO] Failure number 273
[INFO] Failure number 274
[INFO] Failure number 275
[INFO] Failure number 276
[INFO] Failure number 277
[INFO] Failure number 278





array([ 15,  17,  11,  16,  33,  20,  13,  14,  18,  64, 148, 125, 141,
       162, 146, 119, 141, 146, 146, 193, 146, 197, 147, 130, 146, 146,
       146, 185, 184, 177, 169, 164, 176, 198, 171, 176, 176, 196, 164,
       182, 181, 207, 178, 138, 187, 138, 156, 172, 188, 175, 193, 190,
       178, 195, 174, 176, 176, 176, 201, 177, 176, 177, 200, 196, 188,
       150, 156, 150, 150, 149, 150, 149, 150, 186, 210, 190, 193, 202,
       178, 204, 205, 193, 182, 177, 193, 212, 193, 177, 195, 207, 177,
       213, 182, 166, 196, 212, 211, 195, 198, 200, 175, 179, 194, 176,
       198, 207, 192, 196, 193, 195, 197, 190, 193, 184, 207, 197, 196,
       195, 193, 184, 192, 205, 176, 192, 212, 181, 187, 174, 175, 206,
       206, 194, 176, 175, 205, 205, 196, 176, 194, 211, 207, 193, 190,
       209, 205, 191, 212, 185, 186, 205, 189, 212, 186, 176, 198, 210,
       174, 182, 177, 192, 210, 186, 188, 211, 171, 196, 175, 187, 176,
       197, 205, 193, 174, 208, 176, 200, 193, 178, 192, 196, 185, 178,
       176, 175, 202, 202, 195, 199, 191, 201, 187, 180, 213, 205, 208,
       213, 181, 186, 182, 193, 213, 172, 212, 178, 209, 202, 177, 210,
       178, 196, 184, 212, 177, 211, 184, 209, 196, 196, 188, 197, 212,
       210, 199, 196, 192, 174, 176, 207, 196, 181, 193, 178, 193, 186,
       193, 176, 190, 211, 197, 213, 200, 180, 179, 193, 204, 188, 177,
       191, 213, 207, 211, 197, 190, 197, 195, 219, 176, 206, 193, 206,
       192, 208, 208, 176, 176, 193, 194, 176, 191, 179, 197, 173, 188,
       181, 187, 175, 182, 192])

请添加图片描述

从图上结果可以发现，大约50次以后基本收敛

改变np.random.seed

main(seed=1)

[INFO] Failure number 1
[INFO] Failure number 2
[INFO] Failure number 3
[INFO] Failure number 4
[INFO] Failure number 5
[INFO] Failure number 6
[INFO] Failure number 7
[INFO] Failure number 8
[INFO] Failure number 9
[INFO] Failure number 10
[INFO] Failure number 11
[INFO] Failure number 12
[INFO] Failure number 13
[INFO] Failure number 14
[INFO] Failure number 15
[INFO] Failure number 16
[INFO] Failure number 17
[INFO] Failure number 18
[INFO] Failure number 19
[INFO] Failure number 20
[INFO] Failure number 21
[INFO] Failure number 22
[INFO] Failure number 23
[INFO] Failure number 24
[INFO] Failure number 25
[INFO] Failure number 26
[INFO] Failure number 27
[INFO] Failure number 28
[INFO] Failure number 29
[INFO] Failure number 30
[INFO] Failure number 31
[INFO] Failure number 32
[INFO] Failure number 33
[INFO] Failure number 34
[INFO] Failure number 35
[INFO] Failure number 36
[INFO] Failure number 37
[INFO] Failure number 38
[INFO] Failure number 39
[INFO] Failure number 40
[INFO] Failure number 41
[INFO] Failure number 42
[INFO] Failure number 43
[INFO] Failure number 44
[INFO] Failure number 45
[INFO] Failure number 46
[INFO] Failure number 47
[INFO] Failure number 48
[INFO] Failure number 49
[INFO] Failure number 50
[INFO] Failure number 51
[INFO] Failure number 52
[INFO] Failure number 53
[INFO] Failure number 54
[INFO] Failure number 55
[INFO] Failure number 56
[INFO] Failure number 57
[INFO] Failure number 58
[INFO] Failure number 59
[INFO] Failure number 60
[INFO] Failure number 61
[INFO] Failure number 62
[INFO] Failure number 63
[INFO] Failure number 64
[INFO] Failure number 65
[INFO] Failure number 66
[INFO] Failure number 67
[INFO] Failure number 68
[INFO] Failure number 69
[INFO] Failure number 70
[INFO] Failure number 71
[INFO] Failure number 72
[INFO] Failure number 73
[INFO] Failure number 74
[INFO] Failure number 75
[INFO] Failure number 76
[INFO] Failure number 77
[INFO] Failure number 78
[INFO] Failure number 79
[INFO] Failure number 80
[INFO] Failure number 81
[INFO] Failure number 82
[INFO] Failure number 83
[INFO] Failure number 84
[INFO] Failure number 85
[INFO] Failure number 86
[INFO] Failure number 87
[INFO] Failure number 88
[INFO] Failure number 89
[INFO] Failure number 90
[INFO] Failure number 91
[INFO] Failure number 92
[INFO] Failure number 93
[INFO] Failure number 94
[INFO] Failure number 95
[INFO] Failure number 96
[INFO] Failure number 97
[INFO] Failure number 98
[INFO] Failure number 99
[INFO] Failure number 100
[INFO] Failure number 101
[INFO] Failure number 102
[INFO] Failure number 103
[INFO] Failure number 104
[INFO] Failure number 105
[INFO] Failure number 106
[INFO] Failure number 107
[INFO] Failure number 108
[INFO] Failure number 109
[INFO] Failure number 110
[INFO] Failure number 111
[INFO] Failure number 112
[INFO] Failure number 113
[INFO] Failure number 114
[INFO] Failure number 115
[INFO] Failure number 116
[INFO] Failure number 117
[INFO] Failure number 118
[INFO] Failure number 119
[INFO] Failure number 120
[INFO] Failure number 121
[INFO] Failure number 122
[INFO] Failure number 123
[INFO] Failure number 124
[INFO] Failure number 125
[INFO] Failure number 126
[INFO] Failure number 127
[INFO] Failure number 128
[INFO] Failure number 129
[INFO] Failure number 130
[INFO] Failure number 131
[INFO] Failure number 132
[INFO] Failure number 133
[INFO] Failure number 134
[INFO] Failure number 135
[INFO] Failure number 136
[INFO] Failure number 137
[INFO] Failure number 138
[INFO] Failure number 139
[INFO] Failure number 140
[INFO] Failure number 141
[INFO] Failure number 142
[INFO] Failure number 143
[INFO] Failure number 144
[INFO] Failure number 145
[INFO] Failure number 146
[INFO] Failure number 147
[INFO] Failure number 148
[INFO] Failure number 149
[INFO] Failure number 150
[INFO] Failure number 151
[INFO] Failure number 152
[INFO] Failure number 153
[INFO] Failure number 154
[INFO] Failure number 155
[INFO] Failure number 156
[INFO] Failure number 157
[INFO] Failure number 158
[INFO] Failure number 159
[INFO] Failure number 160
[INFO] Failure number 161
[INFO] Failure number 162
[INFO] Failure number 163
[INFO] Failure number 164
[INFO] Failure number 165
[INFO] Failure number 166
[INFO] Failure number 167
[INFO] Failure number 168
[INFO] Failure number 169
[INFO] Failure number 170
[INFO] Failure number 171
[INFO] Failure number 172
[INFO] Failure number 173
[INFO] Failure number 174
[INFO] Failure number 175





array([  16,   16,   41,   16,   31,  263,   13,  322,   41,  445,   33,
        455,  398,  371,  409,  457,  455,  415,  447,  435,  449,  305,
        281,  278,  281,  186,  207,  285,  212,  263,  273,  249,  335,
        301,  211,  316,  276,  476,  268,  477,  442,  338,  356,  169,
        460,  203,  334,  297,  372,  331,  464,  447,  547,  504,  532,
        447,  478,  348,  340,  460,  447,  386, 1082,  484,  348,  407,
        332,  386,  442,  338,  447,  336,  331,  448,  460,  448,  477,
        477,  427,  433,  372,  512, 1832,  407,  331,  406,  378,  442,
        336,  484,  347,  800,  452,  406,  345,  353,  378,  384, 1816,
        464,  399,  698,  512,  570,  331,  335,  376,  334,  532,  348,
        525,  314,  532,  414,  524,  377,  478,  473,  452, 1390,  478,
        427,  407,  334,  478,  800,  800,  470,  532,  374,  452,  508,
        808,  347,  413,  383,  471,  333,  547,  341,  800,  470,  348,
        544,  343,  544,  525,  331,  442,  443,  348,  555,  442,  484,
        471,  469,  341,  544,  439,  452,  471,  331,  402,  659,  348,
        412,  321,  384,  412,  331,  342,  338,  477,  534,  321])

请添加图片描述

main(seed=2)

[INFO] Failure number 1
[INFO] Failure number 2
[INFO] Failure number 3
[INFO] Failure number 4
[INFO] Failure number 5
[INFO] Failure number 6
[INFO] Failure number 7
[INFO] Failure number 8
[INFO] Failure number 9
[INFO] Failure number 10
[INFO] Failure number 11
[INFO] Failure number 12
[INFO] Failure number 13
[INFO] Failure number 14
[INFO] Failure number 15
[INFO] Failure number 16
[INFO] Failure number 17
[INFO] Failure number 18
[INFO] Failure number 19
[INFO] Failure number 20
[INFO] Failure number 21
[INFO] Failure number 22
[INFO] Failure number 23
[INFO] Failure number 24
[INFO] Failure number 25
[INFO] Failure number 26
[INFO] Failure number 27
[INFO] Failure number 28
[INFO] Failure number 29
[INFO] Failure number 30
[INFO] Failure number 31
[INFO] Failure number 32
[INFO] Failure number 33
[INFO] Failure number 34
[INFO] Failure number 35
[INFO] Failure number 36
[INFO] Failure number 37
[INFO] Failure number 38
[INFO] Failure number 39
[INFO] Failure number 40
[INFO] Failure number 41
[INFO] Failure number 42
[INFO] Failure number 43
[INFO] Failure number 44
[INFO] Failure number 45
[INFO] Failure number 46
[INFO] Failure number 47
[INFO] Failure number 48
[INFO] Failure number 49
[INFO] Failure number 50
[INFO] Failure number 51
[INFO] Failure number 52
[INFO] Failure number 53
[INFO] Failure number 54
[INFO] Failure number 55
[INFO] Failure number 56
[INFO] Failure number 57
[INFO] Failure number 58
[INFO] Failure number 59
[INFO] Failure number 60
[INFO] Failure number 61
[INFO] Failure number 62
[INFO] Failure number 63
[INFO] Failure number 64
[INFO] Failure number 65
[INFO] Failure number 66
[INFO] Failure number 67
[INFO] Failure number 68
[INFO] Failure number 69
[INFO] Failure number 70
[INFO] Failure number 71
[INFO] Failure number 72
[INFO] Failure number 73
[INFO] Failure number 74
[INFO] Failure number 75
[INFO] Failure number 76
[INFO] Failure number 77
[INFO] Failure number 78
[INFO] Failure number 79
[INFO] Failure number 80
[INFO] Failure number 81
[INFO] Failure number 82
[INFO] Failure number 83
[INFO] Failure number 84
[INFO] Failure number 85
[INFO] Failure number 86
[INFO] Failure number 87
[INFO] Failure number 88
[INFO] Failure number 89
[INFO] Failure number 90
[INFO] Failure number 91
[INFO] Failure number 92
[INFO] Failure number 93
[INFO] Failure number 94
[INFO] Failure number 95
[INFO] Failure number 96
[INFO] Failure number 97
[INFO] Failure number 98
[INFO] Failure number 99
[INFO] Failure number 100
[INFO] Failure number 101
[INFO] Failure number 102
[INFO] Failure number 103
[INFO] Failure number 104
[INFO] Failure number 105
[INFO] Failure number 106
[INFO] Failure number 107
[INFO] Failure number 108
[INFO] Failure number 109
[INFO] Failure number 110
[INFO] Failure number 111
[INFO] Failure number 112
[INFO] Failure number 113
[INFO] Failure number 114
[INFO] Failure number 115
[INFO] Failure number 116
[INFO] Failure number 117
[INFO] Failure number 118
[INFO] Failure number 119
[INFO] Failure number 120
[INFO] Failure number 121
[INFO] Failure number 122
[INFO] Failure number 123
[INFO] Failure number 124
[INFO] Failure number 125
[INFO] Failure number 126
[INFO] Failure number 127
[INFO] Failure number 128
[INFO] Failure number 129
[INFO] Failure number 130
[INFO] Failure number 131
[INFO] Failure number 132
[INFO] Failure number 133
[INFO] Failure number 134
[INFO] Failure number 135
[INFO] Failure number 136
[INFO] Failure number 137
[INFO] Failure number 138
[INFO] Failure number 139
[INFO] Failure number 140
[INFO] Failure number 141
[INFO] Failure number 142
[INFO] Failure number 143
[INFO] Failure number 144
[INFO] Failure number 145
[INFO] Failure number 146
[INFO] Failure number 147
[INFO] Failure number 148
[INFO] Failure number 149
[INFO] Failure number 150
[INFO] Failure number 151
[INFO] Failure number 152
[INFO] Failure number 153
[INFO] Failure number 154
[INFO] Failure number 155
[INFO] Failure number 156
[INFO] Failure number 157
[INFO] Failure number 158
[INFO] Failure number 159
[INFO] Failure number 160
[INFO] Failure number 161
[INFO] Failure number 162
[INFO] Failure number 163
[INFO] Failure number 164
[INFO] Failure number 165
[INFO] Failure number 166
[INFO] Failure number 167
[INFO] Failure number 168
[INFO] Failure number 169
[INFO] Failure number 170
[INFO] Failure number 171
[INFO] Failure number 172
[INFO] Failure number 173
[INFO] Failure number 174
[INFO] Failure number 175
[INFO] Failure number 176
[INFO] Failure number 177
[INFO] Failure number 178
[INFO] Failure number 179
[INFO] Failure number 180
[INFO] Failure number 181
[INFO] Failure number 182
[INFO] Failure number 183
[INFO] Failure number 184
[INFO] Failure number 185
[INFO] Failure number 186
[INFO] Failure number 187
[INFO] Failure number 188
[INFO] Failure number 189
[INFO] Failure number 190
[INFO] Failure number 191
[INFO] Failure number 192
[INFO] Failure number 193
[INFO] Failure number 194
[INFO] Failure number 195
[INFO] Failure number 196
[INFO] Failure number 197
[INFO] Failure number 198
[INFO] Failure number 199
[INFO] Failure number 200
[INFO] Failure number 201
[INFO] Failure number 202
[INFO] Failure number 203
[INFO] Failure number 204
[INFO] Failure number 205
[INFO] Failure number 206
[INFO] Failure number 207
[INFO] Failure number 208
[INFO] Failure number 209
[INFO] Failure number 210
[INFO] Failure number 211
[INFO] Failure number 212
[INFO] Failure number 213
[INFO] Failure number 214
[INFO] Failure number 215
[INFO] Failure number 216
[INFO] Failure number 217
[INFO] Failure number 218
[INFO] Failure number 219
[INFO] Failure number 220
[INFO] Failure number 221
[INFO] Failure number 222
[INFO] Failure number 223
[INFO] Failure number 224
[INFO] Failure number 225
[INFO] Failure number 226
[INFO] Failure number 227
[INFO] Failure number 228
[INFO] Failure number 229
[INFO] Failure number 230
[INFO] Failure number 231
[INFO] Failure number 232
[INFO] Failure number 233
[INFO] Failure number 234
[INFO] Failure number 235
[INFO] Failure number 236
[INFO] Failure number 237
[INFO] Failure number 238
[INFO] Failure number 239
[INFO] Failure number 240
[INFO] Failure number 241
[INFO] Failure number 242
[INFO] Failure number 243
[INFO] Failure number 244
[INFO] Failure number 245
[INFO] Failure number 246
[INFO] Failure number 247
[INFO] Failure number 248
[INFO] Failure number 249
[INFO] Failure number 250
[INFO] Failure number 251
[INFO] Failure number 252
[INFO] Failure number 253
[INFO] Failure number 254
[INFO] Failure number 255
[INFO] Failure number 256
[INFO] Failure number 257
[INFO] Failure number 258
[INFO] Failure number 259
[INFO] Failure number 260
[INFO] Failure number 261
[INFO] Failure number 262
[INFO] Failure number 263
[INFO] Failure number 264
[INFO] Failure number 265
[INFO] Failure number 266
[INFO] Failure number 267
[INFO] Failure number 268
[INFO] Failure number 269
[INFO] Failure number 270
[INFO] Failure number 271
[INFO] Failure number 272
[INFO] Failure number 273
[INFO] Failure number 274
[INFO] Failure number 275
[INFO] Failure number 276
[INFO] Failure number 277
[INFO] Failure number 278
[INFO] Failure number 279
[INFO] Failure number 280
[INFO] Failure number 281
[INFO] Failure number 282
[INFO] Failure number 283
[INFO] Failure number 284
[INFO] Failure number 285
[INFO] Failure number 286
[INFO] Failure number 287
[INFO] Failure number 288
[INFO] Failure number 289
[INFO] Failure number 290
[INFO] Failure number 291
[INFO] Failure number 292
[INFO] Failure number 293
[INFO] Failure number 294
[INFO] Failure number 295
[INFO] Failure number 296
[INFO] Failure number 297
[INFO] Failure number 298
[INFO] Failure number 299
[INFO] Failure number 300
[INFO] Failure number 301
[INFO] Failure number 302
[INFO] Failure number 303
[INFO] Failure number 304
[INFO] Failure number 305
[INFO] Failure number 306
[INFO] Failure number 307
[INFO] Failure number 308
[INFO] Failure number 309
[INFO] Failure number 310
[INFO] Failure number 311
[INFO] Failure number 312
[INFO] Failure number 313
[INFO] Failure number 314
[INFO] Failure number 315
[INFO] Failure number 316
[INFO] Failure number 317
[INFO] Failure number 318
[INFO] Failure number 319
[INFO] Failure number 320
[INFO] Failure number 321
[INFO] Failure number 322
[INFO] Failure number 323
[INFO] Failure number 324
[INFO] Failure number 325
[INFO] Failure number 326
[INFO] Failure number 327
[INFO] Failure number 328
[INFO] Failure number 329
[INFO] Failure number 330
[INFO] Failure number 331
[INFO] Failure number 332
[INFO] Failure number 333
[INFO] Failure number 334
[INFO] Failure number 335
[INFO] Failure number 336
[INFO] Failure number 337
[INFO] Failure number 338
[INFO] Failure number 339
[INFO] Failure number 340
[INFO] Failure number 341
[INFO] Failure number 342
[INFO] Failure number 343
[INFO] Failure number 344
[INFO] Failure number 345
[INFO] Failure number 346
[INFO] Failure number 347
[INFO] Failure number 348
[INFO] Failure number 349
[INFO] Failure number 350
[INFO] Failure number 351
[INFO] Failure number 352
[INFO] Failure number 353
[INFO] Failure number 354
[INFO] Failure number 355





array([  36,   42,  118,   28,   47,   39,   29,   55,  282,  279,  290,
        265,  273,  137,  168,  157,  312,  172,   19,   19,  150,  157,
        156,  164,  175,  163,  184,  142,  157,  155,  163,  166,  175,
        175,  173,  145,  118,  157,  146,  146,  170,  152,  145,  150,
        174,  153,  165,  151,  152,  173,  150,  150,  172,  176,  168,
        175,  180,  179,  167,  180,  144,  174,  164,  166,  208,  213,
        395,  279,  280,  261,  291,  260,  278,  202,  177,  289,  245,
        235,  781,  292,  278,  174,  215,  139,  289,  298,  186,  188,
        288,  244,  290,  290,  169,  216,  281,  289,  287,  184,  258,
        153,  289,  286,  263,  286,  245,  218,  287,  288,  289,  272,
        158,  301,  285,  248,  260,  282,  221,  206,  427,  266,  206,
        301,  293,  211,  272,  273,  218,  272,  265,  350,  288,  265,
        320,  226,  287,  711,  290,  292,  292,  190,  231,  279,  201,
        293,  282,  281,  296,  547,  256,  302,  283,  287,  211,  246,
        295,  292,  249,  341,  291,  256,  361,  664,  302,  275,  282,
        297,  287,  290,  371,  281,  310,  297,  297,  295,  292,  338,
        343,  298,  287,  278,  308,  286,  458,  273,  333,  307,  290,
        366,  294,  286,  276,  278,  292,  342,  275,  736,  330,  314,
        276,  366,  337,  293,  276,  274,  301,  392,  287,  348,  346,
        276,  276,  307,  424,  288,  299,  363,  292,  294,  276,  189,
        292,  297,  288,  337,  274, 1352,  337,  276,  294,  343,  297,
        337,  302,  303,  342,  276,  399,  319,  287,  295,  278,  335,
        287,  292,  278,  291,  278,  294,  293,  351,  291,  297,  287,
        286,  920,  343,  297,  295,  429,  380,  296,  287,  335,  278,
        350,  387,  278,  288,  287,  397,  292,  298,  348,  232,  320,
        293,  343,  278,  286,  297,  315,  293,  488,  293,  305,  238,
        461,  294,  356,  387,  331,  295,  301,  309,  343,  278,  279,
        392,  279,  305,  295,  292,  295,  331,  278,  271,  292,  381,
        298,  294,  404,  295,  315,  384,  308,  292,  292,  286,  278,
        260,  372,  294,  372,  301,  325,  687,  278,  292,  383,  292,
        284,  278,  234,  295,  401,  292,  300,  279,  278,  404,  415,
        278,  287,  355,  291,  294,  278,  339,  287,  307,  392,  294,
        287,  329,  365])

请添加图片描述

main(seed=3)

[INFO] Failure number 1
[INFO] Failure number 2
[INFO] Failure number 3
[INFO] Failure number 4
[INFO] Failure number 5
[INFO] Failure number 6
[INFO] Failure number 7
[INFO] Failure number 8
[INFO] Failure number 9
[INFO] Failure number 10
[INFO] Failure number 11
[INFO] Failure number 12
[INFO] Failure number 13
[INFO] Failure number 14
[INFO] Failure number 15
[INFO] Failure number 16
[INFO] Failure number 17
[INFO] Failure number 18
[INFO] Failure number 19
[INFO] Failure number 20
[INFO] Failure number 21
[INFO] Failure number 22
[INFO] Failure number 23
[INFO] Failure number 24
[INFO] Failure number 25
[INFO] Failure number 26
[INFO] Failure number 27
[INFO] Failure number 28
[INFO] Failure number 29
[INFO] Failure number 30
[INFO] Failure number 31
[INFO] Failure number 32
[INFO] Failure number 33
[INFO] Failure number 34
[INFO] Failure number 35
[INFO] Failure number 36
[INFO] Failure number 37
[INFO] Failure number 38
[INFO] Failure number 39
[INFO] Failure number 40
[INFO] Failure number 41
[INFO] Failure number 42
[INFO] Failure number 43
[INFO] Failure number 44
[INFO] Failure number 45
[INFO] Failure number 46
[INFO] Failure number 47
[INFO] Failure number 48
[INFO] Failure number 49
[INFO] Failure number 50
[INFO] Failure number 51
[INFO] Failure number 52
[INFO] Failure number 53
[INFO] Failure number 54
[INFO] Failure number 55
[INFO] Failure number 56
[INFO] Failure number 57
[INFO] Failure number 58
[INFO] Failure number 59
[INFO] Failure number 60
[INFO] Failure number 61
[INFO] Failure number 62
[INFO] Failure number 63
[INFO] Failure number 64
[INFO] Failure number 65
[INFO] Failure number 66
[INFO] Failure number 67
[INFO] Failure number 68
[INFO] Failure number 69
[INFO] Failure number 70
[INFO] Failure number 71
[INFO] Failure number 72
[INFO] Failure number 73
[INFO] Failure number 74
[INFO] Failure number 75
[INFO] Failure number 76
[INFO] Failure number 77
[INFO] Failure number 78
[INFO] Failure number 79
[INFO] Failure number 80
[INFO] Failure number 81
[INFO] Failure number 82
[INFO] Failure number 83
[INFO] Failure number 84
[INFO] Failure number 85
[INFO] Failure number 86
[INFO] Failure number 87
[INFO] Failure number 88
[INFO] Failure number 89
[INFO] Failure number 90
[INFO] Failure number 91
[INFO] Failure number 92
[INFO] Failure number 93
[INFO] Failure number 94
[INFO] Failure number 95
[INFO] Failure number 96
[INFO] Failure number 97
[INFO] Failure number 98
[INFO] Failure number 99
[INFO] Failure number 100
[INFO] Failure number 101
[INFO] Failure number 102
[INFO] Failure number 103
[INFO] Failure number 104
[INFO] Failure number 105
[INFO] Failure number 106
[INFO] Failure number 107
[INFO] Failure number 108
[INFO] Failure number 109
[INFO] Failure number 110
[INFO] Failure number 111
[INFO] Failure number 112
[INFO] Failure number 113
[INFO] Failure number 114
[INFO] Failure number 115
[INFO] Failure number 116
[INFO] Failure number 117
[INFO] Failure number 118
[INFO] Failure number 119
[INFO] Failure number 120
[INFO] Failure number 121
[INFO] Failure number 122
[INFO] Failure number 123
[INFO] Failure number 124
[INFO] Failure number 125
[INFO] Failure number 126
[INFO] Failure number 127
[INFO] Failure number 128
[INFO] Failure number 129
[INFO] Failure number 130
[INFO] Failure number 131
[INFO] Failure number 132
[INFO] Failure number 133
[INFO] Failure number 134
[INFO] Failure number 135
[INFO] Failure number 136
[INFO] Failure number 137
[INFO] Failure number 138
[INFO] Failure number 139
[INFO] Failure number 140
[INFO] Failure number 141
[INFO] Failure number 142
[INFO] Failure number 143
[INFO] Failure number 144
[INFO] Failure number 145
[INFO] Failure number 146
[INFO] Failure number 147
[INFO] Failure number 148
[INFO] Failure number 149
[INFO] Failure number 150
[INFO] Failure number 151
[INFO] Failure number 152
[INFO] Failure number 153
[INFO] Failure number 154
[INFO] Failure number 155
[INFO] Failure number 156
[INFO] Failure number 157
[INFO] Failure number 158
[INFO] Failure number 159
[INFO] Failure number 160
[INFO] Failure number 161
[INFO] Failure number 162
[INFO] Failure number 163
[INFO] Failure number 164
[INFO] Failure number 165
[INFO] Failure number 166
[INFO] Failure number 167
[INFO] Failure number 168
[INFO] Failure number 169
[INFO] Failure number 170
[INFO] Failure number 171
[INFO] Failure number 172
[INFO] Failure number 173
[INFO] Failure number 174
[INFO] Failure number 175
[INFO] Failure number 176
[INFO] Failure number 177
[INFO] Failure number 178
[INFO] Failure number 179
[INFO] Failure number 180
[INFO] Failure number 181
[INFO] Failure number 182
[INFO] Failure number 183
[INFO] Failure number 184
[INFO] Failure number 185
[INFO] Failure number 186
[INFO] Failure number 187
[INFO] Failure number 188
[INFO] Failure number 189
[INFO] Failure number 190
[INFO] Failure number 191
[INFO] Failure number 192
[INFO] Failure number 193
[INFO] Failure number 194
[INFO] Failure number 195
[INFO] Failure number 196
[INFO] Failure number 197
[INFO] Failure number 198
[INFO] Failure number 199
[INFO] Failure number 200
[INFO] Failure number 201
[INFO] Failure number 202
[INFO] Failure number 203
[INFO] Failure number 204
[INFO] Failure number 205
[INFO] Failure number 206
[INFO] Failure number 207
[INFO] Failure number 208
[INFO] Failure number 209
[INFO] Failure number 210
[INFO] Failure number 211
[INFO] Failure number 212
[INFO] Failure number 213
[INFO] Failure number 214
[INFO] Failure number 215
[INFO] Failure number 216
[INFO] Failure number 217
[INFO] Failure number 218
[INFO] Failure number 219
[INFO] Failure number 220
[INFO] Failure number 221
[INFO] Failure number 222
[INFO] Failure number 223
[INFO] Failure number 224
[INFO] Failure number 225
[INFO] Failure number 226
[INFO] Failure number 227
[INFO] Failure number 228
[INFO] Failure number 229
[INFO] Failure number 230
[INFO] Failure number 231
[INFO] Failure number 232
[INFO] Failure number 233
[INFO] Failure number 234
[INFO] Failure number 235
[INFO] Failure number 236
[INFO] Failure number 237
[INFO] Failure number 238
[INFO] Failure number 239
[INFO] Failure number 240
[INFO] Failure number 241
[INFO] Failure number 242
[INFO] Failure number 243
[INFO] Failure number 244
[INFO] Failure number 245
[INFO] Failure number 246
[INFO] Failure number 247





array([   31,    16,    20,    64,    52,   117,    39,   117,    21,
          23,   117,    40,   117,   114,    36,    99,    52,    99,
          99,    52,   457,    78,   149,  7380,  6598,  7279,    60,
          38,    58,    58,    58,    58,    98,  7339,   312,   864,
         872,  1118,  6936,  1265,  6731,  2261,  8536,  5051,    60,
        2667,  2835,  6120,   368,  2686,  4655,  3283,  4590,  6095,
        6048,  8916,   363,   339,   314,   861,  2259,   957,  5014,
       10428, 12264,  6073,  6243,  1106,  6216,  6277,   803,  9688,
        6517,   367,   369,   575,   352,   303,  2300,   776,   752,
        2043,  6635,  6416, 12714,   853,  2673,   659,  5926,  2070,
        9343,  1259,  1234,  6241,  1311,   803,  6335, 12721,  3786,
        1260,  6489,   657,  1199,   953,   363,  6335,   369,  2655,
        8548,  1057,  1097,  6489,  9974, 14124,  6577,  6772,  9010,
         367,   369,  1101,  1012,  3394, 18323,  8548,  1453,   386,
        2069,  2835,   882,  8860,  1259,   369,  1079,  8240, 32641,
       10060,   852,   369,  2691,  2277,  6465,   883,  5380,  6413,
       12264,   371,   369, 22763,   369,  6465,  6509,  6335,   752,
        6439,   854, 10060,  2683,  2680,   457, 32656,  7083,  3394,
        3652,  2655,   393,  8442,  2709,   953,  3288,  6408,  2828,
       10085,  6186,   947, 24166,   853,  5031,  9233,  1506, 12731,
        6413,   752,  6462,  2835,  6243,  6517,  1810,   470,  6462,
        6577,  5986,  6243,   470,   953,  1100,   369,  9996,  5380,
         465,   314,  2673,  6524,  1101,  6489, 16853,  6467,   386,
        8548,   613,   853, 32657,  1506,  9688,  2650,  1012,  6268,
         371,  6243,  1234,   369,  6772,   393,  5057, 16853,   905,
        1060,  5966,  6241,  6030,  1012,  6076,   853,  6186, 20635,
         363,   657,   363,  1138, 23114,   339,   613,  9688,   411,
        6076,  3652,   369,  6509])

请添加图片描述

可以发现，随着seed的变化，也就是随机数的变化，从红线中看出，收敛的速度基本相同，但是最终收敛到的维持单摆不倒的时长，会有比较明显的变化，说明系统仍旧有提高的空间。

ML--小小白
关注
专栏目录
强化学习Reinforcement Learning对抗性训练与环境适应性
AI天才研究院
07-23 691
强化学习Reinforcement Learning对抗性训练与环境适应性 作者：禅与计算机程序设计艺术 / Zen and the Art of Computer Programming 关键词：强化学习，对抗性训练，环境适应性，Q-learning，深度强化学习，环境模拟
强化学习Reinforcement Learning在智慧城市中的应用案例
AGI通用人工智能之禅
07-24 37
强化学习Reinforcement Learning在智慧城市中的应用案例 作者：禅与计算机程序设计艺术 / Zen and the Art of Computer Programming 关键词：强化学习
CS229课程讲义及作业-Andrew Ng
11-21
CS229课程讲义及作业-Andrew NgCS229课程讲义及作业-Andrew NgCS229课程讲义及作业-Andrew NgCS229课程讲义及作业-Andrew Ng
吴恩达笔记——强化学习
weixin_72100405的博客
04-28 402
第一步是随机产生一个Q，用来得到作为trainingset的x和y，然后把x和对应的y输入到NN中。第三种：mini batch训练 减少训练时间 ——这个也可以作为普通NN的训练方法以减小训练时间。第一种：把输出层从一个神经元变成跟action个数一样的神经元数量。epsilon的尝试要从高开始逐渐降低，比如从1到0.01。
吴恩达机器学习 第三课 week3 强化学习（月球着陆器自动着陆）
weixin_43490087的博客
07-06 1211
Coursera课程 吴恩达机器学习 第3课 ：无监督学习、推荐算法和强化学习
斯坦福吴恩达《机器学习》--增强学习
zombee0的博客
01-21 3362
在监督学习中，算法试图模仿训练机的labels y,训练集中的每一个输入x都有一个确定的对应的y，但是对于很多需要连续作决定的问题和控制问题，给算法提供一个明确的标签是很难的。例如我们有一个四足机器人，并且试图让他行走，开始的时候我们并不知道采取怎样的操作使他行走，也不知道怎么给算法提供一个标签来模仿。   在增强学习中，我们会给算法提供一个奖励函数来反应做的好还是不好。
cs229 作业答案
07-23
stanford 吴恩达机器学习 cs229作业代码 ipynb格式 内含各个作业文件
吴恩达来信：强化学习的发展！
weixin_40920183的博客
08-09 127
Dear friends,(中文版见下文)While working on Course 3 of theMachine Learning Specialization, which covers reinforcement learning, I was reflecting on how reinforcement learning algorithms are...
增强学习（一）——马尔科夫决策过程（MDP）
BertDai的博客
08-02 4948
增强学习（reinforcement learning,RL）是近年来机器学习和智能控制领域的主要方法之一。在增强学习中有三个概念：状态、动作和回报。 “状态（state）”是描述当前情况的。对一个正在学习行走的机器人来说，状态是它的两条腿的位置。对一个围棋程序来说，状态是棋盘上所有棋子的位置。 “动作（action）”是一个智能体在每个状态中可以做的事情。给定一个机器人两条腿的状态或位置，它可以在一定距离内走几步。通常一个智能体只能采取有限或者固定范围内的动作。例如一个机器人的
一切皆是映射：强化学习的基础概念与核心算法
最新发布
AI天才研究院
07-28 256
一切皆是映射：强化学习的基础概念与核心算法 作者：禅与计算机程序设计艺术 / Zen and the Art of Computer Programming 关键词： 强化学习、策略学习、智能体、环境、奖励、价值函数、策略迭代、Q学习、深度Q网络
强化学习：在音乐生成中的应用
AGI通用人工智能之禅
06-21 61
强化学习：在音乐生成中的应用 1. 背景介绍 随着人工智能技术的不断发展,音乐生成已成为一个备受关注的研究领域。传统的音乐创作过程通常需要作曲家投入大量的时间和精力,而利用人工智能技术生成音乐则可以极大地提高效率,并为音乐创作带来新的可能性。在这一背景下,强化学习作为一种重要的机器学习方法,为
cs229 2017 作业答案
03-21
cs229 机器学习
斯坦福吴恩达强化学习例程
05-31
斯坦福公开课吴恩达老师提供的强化学习例程，比较好理解，Q学习的例程
CS229所有讲义+作业+作业讲解
02-08
Andrew Ng教授的机器学习课程cs229对应的上课讲义，作业，以及作业讲解
斯坦福机器学习公开课CS229讲义作业及matlab代码资料
05-19
个人整理 斯坦福公开课 机器学习CS229课程 较全讲义、作业和matlab代码。
斯坦福大学CS229机器学习完整详细笔记 中文版 （含Coursera课程作业代码 以及全套中文版笔记）.zip
05-12
包含斯坦福大学课程笔记中文版 Coursera课程笔记以及作业答案matlab 完整版
【吴恩达来信】强化学习的发展！
weixin_43509834的博客
08-08 283
亲爱的朋友们，在准备学习机器学习专业课程的第三课时（其中包括强化学习的内容），我认真思考了为何强化学习算法在使用上仍是非常苛刻的。它们对超参数的选择非常敏感，有超参数调优经验的人可能会获得10倍或100倍的性能提升。十年前，使用有监督深度学习算法同样苛刻，但随着建立有监督模型的系统方法的研究不断进步，它也逐渐变得更加稳健。强化学习算法在未来十年还会变得更加稳健吗？我希望如此。然而，强化学习在建立真实世界（非模拟）基准方面面临着一个独特的障碍。............
CS229 吴恩达机器学习 习题大作业答案 problem sets 04 PS04（第一问，欢迎指教）Neural Networks: MNIST image classification
qq_26928055的博客
04-29 1969
1. Neural Networks: MNIST image classification import numpy as np import matplotlib.pyplot as plt import math MAX_POOL_SIZE = 5 CONVOLUTION_SIZE = 4 CONVOLUTION_FILTERS = 2 def forward_softmax(x): """ Compute softmax function for a single example
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

ML--小小白

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值