机器学习：使用matlab实现SVM完成垃圾邮件识别

预处理

在开始机器学习之前，多观察数据集中的数据通常很有帮助。比如在下面这封邮件里

我们可以看到一个 URL、一个电子邮件地址（在末尾）、数字和美元金额。

虽然许多电子邮件会包含类似类型的文本（例如，数字、其他 URL 或其他电子邮件地址），但几乎每封电子邮件中的这些文本都会有所不同。

因此，处理电子邮件时常用的一种方法是将这些值“标准化”，以便所有 URL 都被视为相同，所有数字都被视为相同等。例如，我们可以将电子邮件中的每个 URL 替换为唯一的字符串“httpaddr”表示存在 URL。这具有让垃圾邮件分类器根据是否存在任何 URL 而不是特定 URL 是否存在来做出分类决定的效果。这通常会提高垃圾邮件分类器的性能，因为垃圾邮件发送者通常会随机化 URL，因此在新的垃圾邮件中再次看到任何特定 URL 的几率非常小。

在 processEmail.m 中，我们实现了以下电子邮件预处理和规范化步骤：

• 小写：将整个电子邮件转换为小写，因此忽略大写（例如，将 IndIcaTE 视为与指示相同）。
• 剥离 HTML：从电子邮件中删除所有 HTML 标记。许多电子邮件通常带有 HTML 格式。我们删除了所有的 HTML 标记，因此只保留了内容。
• 规范化 URL：所有 URL 都替换为文本“httpaddr”。
• 标准化电子邮件地址：所有电子邮件地址都替换为文本“emailaddr”。
• 规范化数字：所有数字都替换为文本“数字”。
• 标准化美元：所有美元符号 ($) 都替换为文本“美元”。
• 词干：词被简化为词干形式。例如，‘discount’、‘discounts’、‘discounted’和’discounting’都替换为’discount’。有时，Stemmer 实际上会从末尾剥离额外的字符，因此“包含”、“包含”、“包含”和“包含”都被替换为“包含”。
• 去除非单词：去除非单词和标点符号。所有空格（制表符、换行符、空格）都已被修剪为单个空格字符。

词库映射

另外，我们还需要将数据中的单词替换为数字，即用单词在我们词库里的索引来替代字符串。字典库包含所有信件中出现次数超过100的单词（如果囊括太多词汇，包括那些仅出现过几次的，很可能会出现过拟合），共1899个。

在 MATLAB 中，可以使用 strcmp 函数比较两个字符串。例如，strcmp(str1, str2)仅当两个字符串相等时才会返回 1。在提供的起始代码中，vocabList 是一个包含词汇表中单词的cell-array。在 MATLAB 中，除了它的元素也可以是字符串（它们不能在普通的 MATLAB 矩阵/向量中），cell-array就像一个普通数组（即向量），可以大括号对它们进行索引。

前两个部分的函数如下：

function word_indices = processEmail(email_contents)
%PROCESSEMAIL preprocesses a the body of an email and
%returns a list of word_indices 
%   word_indices = PROCESSEMAIL(email_contents) preprocesses 
%   the body of an email and returns a list of indices of the 
%   words contained in the email. 
%

% Load Vocabulary
vocabList = getVocabList();

% Init return value
word_indices = [];

% ========================== Preprocess Email ===========================

% Find the Headers ( \n\n and remove )
% Uncomment the following lines if you are working with raw emails with the
% full headers

% hdrstart = strfind(email_contents, ([char(10) char(10)]));
% email_contents = email_contents(hdrstart(1):end);

% Lower case
email_contents = lower(email_contents);

% Strip all HTML
% Looks for any expression that starts with < and ends with > and replace
% and does not have any < or > in the tag it with a space
email_contents = regexprep(email_contents, '<[^<>]+>', ' ');

% Handle Numbers
% Look for one or more characters between 0-9
email_contents = regexprep(email_contents, '[0-9]+', 'number');

% Handle URLS
% Look for strings starting with http:// or https://
email_contents = regexprep(email_contents, ...
                           '(http|https)://[^\s]*', 'httpaddr');

% Handle Email Addresses
% Look for strings with @ in the middle
email_contents = regexprep(email_contents, '[^\s]+@[^\s]+', 'emailaddr');

% Handle $ sign
email_contents = regexprep(email_contents, '[$]+', 'dollar');


% ========================== Tokenize Email ===========================

% Output the email to screen as well
fprintf('\n==== Processed Email ====\n\n');

% Process file
l = 0;

while ~isempty(email_contents)

    % Tokenize and also get rid of any punctuation
    [str, email_contents] = ...
       strtok(email_contents, ...
              [' @$/#.-:&*+=[]?!(){},''">_<;%' char(10) char(13)]);
   
    % Remove any non alphanumeric characters
    str = regexprep(str, '[^a-zA-Z0-9]', '');

    % Stem the word 
    % (the porterStemmer sometimes has issues, so we use a try catch block)
    try str = porterStemmer(strtrim(str)); 
    catch str = ''; continue;
    end;

    % Skip the word if it is too short
    if length(str) < 1
       continue;
    end

    % Look up the word in the dictionary and add to word_indices if
    % found
    % ====================== YOUR CODE HERE ======================
    % Instructions: Fill in this function to add the index of str to
    %               word_indices if it is in the vocabulary. At this point
    %               of the code, you have a stemmed word from the email in
    %               the variable str. You should look up str in the
    %               vocabulary list (vocabList). If a match exists, you
    %               should add the index of the word to the word_indices
    %               vector. Concretely, if str = 'action', then you should
    %               look up the vocabulary list to find where in vocabList
    %               'action' appears. For example, if vocabList{18} =
    %               'action', then, you should add 18 to the word_indices 
    %               vector (e.g., word_indices = [word_indices ; 18]; ).
    % 
    % Note: vocabList{idx} returns a the word with index idx in the
    %       vocabulary list.
    % 
    % Note: You can use strcmp(str1, str2) to compare two strings (str1 and
    %       str2). It will return 1 only if the two strings are equivalent.
    %
    
    idx=find(strcmp(str,vocabList));
    word_indices=[word_indices;idx];
    
    % =============================================================


    % Print to screen, ensuring that the output lines are not too long
    if (l + length(str) + 1) > 78
        fprintf('\n');
        l = 0;
    end
    fprintf('%s ', str);
    l = l + length(str) + 1;

end

% Print footer
fprintf('\n\n=========================\n');

end

做完前两步处理，我们的信件就从一封信变成一个数字向量了。

%% Initialization
clear;

% Extract Features
file_contents = readFile('emailSample1.txt');
word_indices  = processEmail(file_contents);
% Print Stats
disp(word_indices)

构造特征向量

对于一封信，它的特征向量就是词库里的某个词有没有在这封信里出现过，有则该项为1，没有则是0，很简单：

function x = emailFeatures(word_indices)
%EMAILFEATURES takes in a word_indices vector and produces a feature vector
%from the word indices
%   x = EMAILFEATURES(word_indices) takes in a word_indices vector and 
%   produces a feature vector from the word indices. 

% Total number of words in the dictionary
n = 1899;

% You need to return the following variables correctly.
x = zeros(n, 1);

% ====================== YOUR CODE HERE ======================
% Instructions: Fill in this function to return a feature vector for the
%               given email (word_indices). To help make it easier to 
%               process the emails, we have have already pre-processed each
%               email and converted each word in the email into an index in
%               a fixed dictionary (of 1899 words). The variable
%               word_indices contains the list of indices of the words
%               which occur in one email.
% 
%               Concretely, if an email has the text:
%
%                  The quick brown fox jumped over the lazy dog.
%
%               Then, the word_indices vector for this text might look 
%               like:
%               
%                   60  100   33   44   10     53  60  58   5
%
%               where, we have mapped each word onto a number, for example:
%
%                   the   -- 60
%                   quick -- 100
%                   ...
%
%              (note: the above numbers are just an example and are not the
%               actual mappings).
%
%              Your task is take one such word_indices vector and construct
%              a binary feature vector that indicates whether a particular
%              word occurs in the email. That is, x(i) = 1 when word i
%              is present in the email. Concretely, if the word 'the' (say,
%              index 60) appears in the email, then x(60) = 1. The feature
%              vector should look like:
%
%              x = [ 0 0 0 0 1 0 0 0 ... 0 0 0 0 1 ... 0 0 0 1 0 ..];
%
%

for i=1:length(word_indices)
    x(word_indices(i))=1;
end

% =========================================================================
    

end

训练

让数据中的每封信都经过上面的处理之后，我们就可以进行常规的SVM训练了：

% Load the Spam Email dataset
% You will have X, y in your environment
load('spamTrain.mat');%这里加载的是已经处理好的数据
C = 0.1;
model = svmTrain(X, y, C, @linearKernel);

p = svmPredict(model, X);
fprintf('Training Accuracy: %f\n', mean(double(p == y)) * 100);

% Load the test dataset
% You will have Xtest, ytest in your environment
load('spamTest.mat');

p = svmPredict(model, Xtest);
fprintf('Test Accuracy: %f\n', mean(double(p == ytest)) * 100);

最后得到的训练集准确率$99.85\%$，测试集准确度$98.9\%$

通过观察模型的参数，我们可以看看包含什么词的更容易被我们的分类器当作垃圾邮件：

% Sort the weights and obtin the vocabulary list
[weight, idx] = sort(model.w, 'descend');
vocabList = getVocabList();
for i = 1:15
    if i == 1
        fprintf('Top predictors of spam: \n');
    end
    fprintf('%-15s (%f) \n', vocabList{idx(i)}, weight(i));
end