AI6012 Java/c++

Java Python AI6012: Machine Learning Methodologies &
Applications Assignment (25 points)
Important notes: to ffnish this assignment, you are allowed to look up textbooks or
search materials via Google for reference. NO plagiarism from classmates is allowed.
The submission deadline is by 11:59 pm, Sept. 30, 2022. The ffle to be submitted
is a single PDF (no source codes are required to be submitted). Multiple submission
attempts are allowed, and the last one will be graded. A submission link is available
under “Assignments” of the course website in NTULearn.
Question 1 (10 marks): Consider a multi-class classiffcation problem of C classes.
Based on the parametric forms of the conditional probabilities of each class introduced
on the 39th Page (“Extension to Multiple Classes”) of the lecture notes of L4, derive
the learning procedure of regularized logistic regression for multi-class classiffcation
problems.
Hint: deffne a loss function by borrowing an idea from binary classiffcation, and
derive the gradient descent rules to update {w(c)}’s.
Question 2 (5 marks): This is a hands-on exercise to use the SVC API of scikitlearn
1
to
train a SVM with the linear kernel and the rbf kernel, respectively, on a binary
classiffcation dataset. The details of instructions are described as follows.
1. Download the a9a dataset from the LIBSVM Dataset page.
This is a preprocessed dataset of the Adult dataset in the UCI Irvine Machine
Learning Repository
2
, which consists of a training set (available here) and a test
set (available here).
Each ffle (the train set or the test set) is a text format in which each line represents
a labeled data instance as follows:
label index1:value1 index2:value2 ...
where “label” denotes the class label of each instance, “indexT” denotes the
T-th feature, and valueT denotes the value of the T-th feature of the instance.
1Read Pages 63-64 of the lecture notes of L5 for reference
2The details of the original Adult dataset can be found here.
1This is a sparse format, where only non-zero feature values are stored for each
instance. For example, suppose given a data set, where each data instance has 5
dimensions (features). If a data instance whose label is “+1” and the input data
instance vector is [2 0 2.5 4.3 0], then it is presented in a line as
+1 1:2 3:2.5 4:4.3
Hint: sciki-learn provides an API (“sklearn.datasets.load svmlight ffle”) to load
such a sparse data forma AI6012、Java/c++ t. Detailed information is available here.
2. Regarding the linear kernel, show 3-fold cross-validation results in terms of classiffcation
accuracy on the training set with different values of the parameter C in
{0.01, 0.05, 0.1, 0.5, 1}, respectively, in the following table. Note that for all the
other parameters, you can simply use the default values or specify the speciffc
values you used in your submitted PDF ffle.
Table 1: The 3-fold cross-validation results of varying values of C in SVC with linear
kernel on the a9a training set (in accuracy).
C = 0.01 C = 0.05 C = 0.1 C = 0.5 C = 1
? ? ? ? ?
3. Regarding the rbf kernel, show 3-fold cross-validation results in terms of classiffcation
accuracy on the training set with different values of the parameter gamma
(i.e., σ
2 on the lecture notes) in {0.01, 0.05, 0.1, 0.5, 1} and different values of
the parameter C in {0.01, 0.05, 0.1, 0.5, 1}, respectively, in the following table.
Note that for all the other parameters, you can simply use the default values or
specify the speciffc values you used in your submitted PDF ffle.
Table 2: The 3-fold cross-validation results of varying values of gamma and C in SVC
with rbf kernel on the a9a training set (in accuracy).
Hint: there are no speciffc APIs that integrates cross-validation into SVMs in
sciki-learn. However, you can use some APIs under the category “Model Selection
→ Model validation” to implement it. Some examples can be found here.
4. Based on the results shown in Tables 1-2, determine the best kernel and the best
parameter setting. Use the best kernel with the best parameter setting to train a
SVM using the whole training set and make predictions on test set to generate
the following table:
2Table 3: Test results of SVC on the a9a test set (in accuracy).
Specify which kernel with what parameter setting
Accuracy of SVMs ?
Question 3 (5 marks): The optimization problem of linear soft-margin SVMs can
be re-formulated as an instance of empirical structural risk minimization (refer to Page
37 on L5 notes). Show how to reformulate it. Hint: search reference about the hinge
loss.
Question 4 (5 marks): Using the kernel trick introduced in L5 to extend the regularized
linear regression model (L3) to solve nonlinear regression problems. Derive a
closed-form solution (i.e., to derive a kernelized version of the closed-form solution on
Page 50 of L3)