Exercise 4: Logistic Regressionand Newton's Method

转载 2011年10月16日 12:43:43

Raw page: http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&doc=exercises/ex4/ex4.html

Exercise 4: Logistic Regressionand Newton's Method

In this exercise, you will use Newton's Method to implementlogistic regression on a classification problem.


To begin, download ex4Data.zipand extract the files from the zip file.

For this exercise, suppose that a high school has a datasetrepresenting 40 students who were admitted to college and 40 students who werenot admitted. Each  trainingexample contains a student's score on two standardized exams and a label ofwhether the student was admitted.

Your task is to build a binary classification model thatestimates college admission chances based on a student's scores on two exams.In your training data,

a. The first column of yourx array represents all Test 1 scores, and the second column represents all Test2 scores.

b. The y vector uses '1' tolabel a student who was admitted and '0' to label a student who was notadmitted.


Plot the data

Load the data for the training examples into your programand add the intercept term into yourx matrix.

Before beginning Newton's Method, we will first plot thedata using different symbols to represent the two classes. In Matlab/Octave,you can separate the positive class and the negative class using the findcommand:

% find returns the indices of the
% rows meeting the specified condition
pos = find(y == 1); neg = find(y == 0);
% Assume the features are in the 2nd and 3rd
% columns of x
plot(x(pos, 2), x(pos,3), '+'); hold on
plot(x(neg, 2), x(neg, 3), 'o')

Your plot should look like the following:

Newton's Method

Recall that in logistic regression, the hypothesis functionis





In our example, the hypothesis is interpreted as theprobability that a driver will be accident-free, given the values of thefeatures in x.

Matlab/Octave does not have a library function for thesigmoid, so you will have to define it yourself. The easiest way to do this isthrough an inline expression:

g = inline('1.0 ./ (1.0 + exp(-z))'); 
% Usage: To find the value of the sigmoid 
% evaluated at 2, call g(2)

The cost function is defined as


Our goal is to use Newton's method to minimize thisfunction. Recall that the update rule for Newton's method is


In logistic regression, the gradient and the Hessian are



Note that the formulas presented above are the vectorizedversions. Specifically, this means that , ,while and are scalars.


Now, implement Newton's Method in your program, startingwith the initial value of . To determinehow many iterations to use, calculate for each iterationand plot your results as you did in Exercise 2. As mentioned in thelecture videos, Newton's method often converges in 5-15 iterations. If you findyourself using far more iterations, you should check for errors in yourimplementation.

After convergence, use your values of theta to find thedecision boundary in the classification problem. The decision boundary isdefined as the line where

which corresponds to

Plotting the decision boundary is equivalent to plottingthe line. When youare finished, your plot should appear like the figure below.


Finally, record your answers to these questions.

1. What values of did you get? How manyiterations were required for convergence?

2. What is the probabilitythat a student with a score of 20 on Exam 1 and a score of 80 onExam 2 will not be admitted?


After you have completed the exercises above, please referto the solutions below and check that your implementation and your answers arecorrect. In a case where your implementation does not result in the sameparameters/phenomena as described below, debug your solution until you manageto replicate the same effect as our implementation.

A complete m-file implementation of the solutions can befound here.

Newton's Method

1. Your final values oftheta should be





Plot. Your plot of the costfunction should look similar to the picture below:

From this plot, you can infer that Newton's Method hasconverged by around 5 iterations. In fact, by looking at a printout of thevalues of J, you will see that J changes by less than between the 4th and 5thiterations. Recall that in the previous two exercises, gradient descent tookhundreds or even thousands of iterations to converge. Newton's Method is muchfaster in comparison.

2. The probability that astudent with a score of 20 on Exam 1 and 80 on Exam 2 will not beadmitted to college is 0.668.


Exercise 4: Logistic Regression and Newton's Method

In this exercise, you will use Newton's Method to implement logistic regression on a classification ...
  • ytlcainiao
  • ytlcainiao
  • 2013年05月25日 08:29
  • 1241

Logistic regression ,Softmax regression and Newton's method

首先说的是Logistic regression,z这个回归的相对还是比较简单的,但是一般来说他只能用于分两类(0或1)的情况,虽然我曾在Andrew Ng的课上好像记得也可以处理多类的情况,但是 S...
  • u012303532
  • u012303532
  • 2015年04月12日 21:04
  • 601

使用牛顿迭代方法(Newton’s method)来估计方程的解

  • Allenalex
  • Allenalex
  • 2013年11月18日 16:33
  • 2427

Logistic Regression and Newton's Method Exercise

本文将阐述logistic regression 的原理及编程实现。理论参考文档:http://openclassroom.stanford.edu/MainFolder/DocumentPage.p...
  • u010457543
  • u010457543
  • 2015年08月28日 20:22
  • 323

[Exercise 3] Logistic Regression and Newton's Method

这个练习将通过牛顿方法来实现逻辑回归分类。Dataex4Data.zip这里给出的训练样本的特征为80个学生的两门功课的分数. 样本值为对应的同学是否允许被上大学,如果是允许的话则用’1’表示,否则不...
  • sherlockzoom
  • sherlockzoom
  • 2015年07月15日 11:30
  • 526

Exercise: Logistic Regression and Newton's Method

 Exercise: Logistic Regression and Newton's Method 题目地址: Exercise: Logistic Regression 题...
  • starzhou
  • starzhou
  • 2016年05月31日 15:18
  • 189

Newton's method in Matlab

举例说明syms x y; % Make x,y symbolic f1 = x^2 + y^3 - 1; % Make your two equations (from your example) ...
  • ziqian0512
  • ziqian0512
  • 2014年10月23日 07:48
  • 1030

[机器学习实验3]Logistic Regression and Newton Method

第三篇实验记录,logic回归的,一个二分类问题,输入变量的维度比较低(二维),所以可以在图像画出边界线来直观感受的。 题目链接: http://openclassroom.stanford....
  • gyh_420
  • gyh_420
  • 2017年09月02日 20:20
  • 98

Logistic Regression and Newton's Method

close all,clear,clc x = load('ex4x.dat'); y = load('ex4y.dat'); % find returns the indices of the ...
  • suky520
  • suky520
  • 2013年12月22日 11:36
  • 780

Matlab Newton‘s method

定义函数 function y=f(x) y=f(x);%函数f(x)的表达式 end function z=h(x) z=h(x);%函数h(x)的表达式 end...
  • u013152895
  • u013152895
  • 2015年03月16日 09:34
  • 797
您举报文章:Exercise 4: Logistic Regressionand Newton's Method