2020 cs231n 作业3 笔记 NetworkVisualization-PyTorch

NetworkVisualization-PyTorch

Saliency Maps

论文链接，Saliency Maps用来表示每个像素对图像分类得分的影响程度。这里是通过反向传播，来计算每个图片像素的梯度的绝对值，然后在三个通道中选最大值，图片像素维度为（3,H,W），则Saliency Maps的维度为（H,W）

def compute_saliency_maps(X, y, model):
    """
    Compute a class saliency map using the model for images X and labels y.

    Input:
    - X: Input images; Tensor of shape (N, 3, H, W)
    - y: Labels for X; LongTensor of shape (N,)
    - model: A pretrained CNN that will be used to compute the saliency map.

    Returns:
    - saliency: A Tensor of shape (N, H, W) giving the saliency maps for the input
    images.
    """
    # Make sure the model is in "test" mode
    model.eval()

    # Make input tensor require gradient
    X.requires_grad_()

    saliency = None
    ##############################################################################
    # TODO: Implement this function. Perform a forward and backward pass through #
    # the model to compute the gradient of the correct class score with respect  #
    # to each input image. You first want to compute the loss over the correct   #
    # scores (we'll combine losses across a batch by summing), and then compute  #
    # the gradients with a backward pass.                                        #
    ##############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    model.zero_grad()
    loss_fun = torch.nn.CrossEntropyLoss()
    y_pred = model(X)
    loss = loss_fun(y_pred, y)
    loss.backward()
    saliency,_ = X.grad.abs().max(axis=1)

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ##############################################################################
    #                             END OF YOUR CODE                               #
    ##############################################################################
    return saliency

输出结果：

Fooling Images

利用一张原有的图片，给定一个目标分类，通过对该图片在目标分类上的得分进行梯度上升，或者对该图片在目标分类上的loss进行梯度下降，来修改原有图片，使其在目标分类上的得分最高（loss最低）。原有图片和修改后的图片人眼一般看不出差别。

def make_fooling_image(X, target_y, model):
    """
    Generate a fooling image that is close to X, but that the model classifies
    as target_y.

    Inputs:
    - X: Input image; Tensor of shape (1, 3, 224, 224)
    - target_y: An integer in the range [0, 1000)
    - model: A pretrained CNN

    Returns:
    - X_fooling: An image that is close to X, but that is classifed as target_y
    by the model.
    """
    # Initialize our fooling image to the input image, and make it require gradient
    X_fooling = X.clone()
    X_fooling = X_fooling.requires_grad_()

    learning_rate = 1
    ##############################################################################
    # TODO: Generate a fooling image X_fooling that the model will classify as   #
    # the class target_y. You should perform gradient ascent on the score of the #
    # target class, stopping when the model is fooled.                           #
    # When computing an update step, first normalize the gradient:               #
    #   dX = learning_rate * g / ||g||_2                                         #
    #                                                                            #
    # You should write a training loop.                                          #
    #                                                                            #
    # HINT: For most examples, you should be able to generate a fooling image    #
    # in fewer than 100 iterations of gradient ascent.                           #
    # You can print your progress over iterations to check your algorithm.       #
    ##############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    model.eval()
    iters =100
    y = torch.LongTensor([target_y])
    loss_fun = torch.nn.CrossEntropyLoss()
    for i in range(iters):
      print('i',i)
      score = model(X_fooling)
      print('score',score.argmax(axis=1))
      print('y',y)
      if score.argmax(axis=1) == y:
        break
      loss = loss_fun(score, y)
      model.zero_grad()
      loss.backward()
      with torch.no_grad():
        g = X_fooling.grad
        dX = learning_rate * g / torch.norm(g)
        X_fooling -= dX
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ##############################################################################
    #                             END OF YOUR CODE                               #
    ##############################################################################
    return X_fooling

输出：

原来图片分类是干草，修改后的图片分类变成了黄貂鱼

Class visualization

和Fooling Images有点类似，不过把开始的图片换成了一个随机的噪声图片。给定一个目标分类，通过对该图片在目标分类上的得分进行梯度上升，或者对该图片在目标分类上的loss进行梯度下降，来修改原有图片，使其在目标分类上的得分最高（loss最低）

def class_visualization_update_step(img, model, target_y, l2_reg, learning_rate):
    ########################################################################
    # TODO: Use the model to compute the gradient of the score for the     #
    # class target_y with respect to the pixels of the image, and make a   #
    # gradient step on the image using the learning rate. Don't forget the #
    # L2 regularization term!                                              #
    # Be very careful about the signs of elements in your code.            #
    ########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    
    model.eval()  
    score = model(img)
    sy = score[:,target_y]
    model.zero_grad()
    sy.backward()
    dimg = img.grad + 2 * l2_reg * img
    with torch.no_grad():
      img += learning_rate * dimg / torch.norm(dimg)
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ########################################################################
    #                             END OF YOUR CODE                         #
    ########################################################################

输出：