Numpy实现DecisionTree(1)

最后

Python崛起并且风靡，因为优点多、应用领域广、被大牛们认可。学习 Python 门槛很低，但它的晋级路线很多，通过它你能进入机器学习、数据挖掘、大数据，CS等更加高级的领域。Python可以做网络应用，可以做科学计算，数据分析，可以做网络爬虫，可以做机器学习、自然语言处理、可以写游戏、可以做桌面应用…Python可以做的很多，你需要学好基础，再选择明确的方向。这里给大家分享一份全套的 Python 学习资料，给那些想学习 Python 的小伙伴们一点帮助！

👉Python所有方向的学习路线👈

Python所有方向的技术点做的整理，形成各个领域的知识点汇总，它的用处就在于，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。

👉Python必备开发工具👈

工欲善其事必先利其器。学习Python常用的开发软件都在这里了，给大家节省了很多时间。

👉Python全套学习视频👈

我们在看视频学习的时候，不能光动眼动脑不动手，比较科学的学习方法是在理解之后运用它们，这时候练手项目就很适合了。

👉实战案例👈

学python就与学数学一样，是不能只看书不做题的，直接看步骤和答案会让人误以为自己全都掌握了，但是碰到生题的时候还是会一筹莫展。

因此在学习python的过程中一定要记得多动手写代码，教程只需要看一两遍即可。

👉大厂面试真题👈

我们学习Python必然是为了找到高薪的工作，下面这些面试题是来自阿里、腾讯、字节等一线互联网大厂最新的面试资料，并且有阿里大佬给出了权威的解答，刷完这一套面试资料相信大家都能找到满意的工作。

需要这份系统化学习资料的朋友，可以戳这里获取

一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！

Xy = np.concatenate((X, y), axis=1)

n_samples, n_features = np.shape(X)

if n_samples >= self.min_samples_split and current_depth <= self.max_depth:

Calculate the impurity for each feature

for feature_i in range(n_features):

All values of feature_i

feature_values = np.expand_dims(X[:, feature_i], axis=1)

unique_values = np.unique(feature_values)

Iterate through all unique values of feature column i and

calculate the impurity

for threshold in unique_values:

Divide X and y depending on if the feature value of X at index feature_i

meets the threshold

Xy1, Xy2 = divide_on_feature(Xy, feature_i, threshold)

if len(Xy1) > 0 and len(Xy2) > 0:

Select the y-values of the two sets

y1 = Xy1[:, n_features:]

y2 = Xy2[:, n_features:]

Calculate impurity

impurity = self._impurity_calculation(y, y1, y2)

If this threshold resulted in a higher information gain than previously

recorded save the threshold value and the feature

index

if impurity > largest_impurity:

largest_impurity = impurity

best_criteria = {“feature_i”: feature_i, “threshold”: threshold}

best_sets = {

“leftX”: Xy1[:, :n_features], # X of left subtree

“lefty”: Xy1[:, n_features:], # y of left subtree

“rightX”: Xy2[:, :n_features], # X of right subtree

“righty”: Xy2[:, n_features:] # y of right subtree

}

if largest_impurity > self.min_impurity:

Build subtrees for the right and left branches

true_branch = self._build_tree(best_sets[“leftX”], best_sets[“lefty”], current_depth + 1)

false_branch = self._build_tree(best_sets[“rightX”], best_sets[“righty”], current_depth + 1)

return DecisionNode(feature_i=best_criteria[“feature_i”], threshold=best_criteria[

“threshold”], true_branch=true_branch, false_branch=false_branch)

We’re at leaf => determine value

leaf_value = self._leaf_value_calculation(y)

return DecisionNode(value=leaf_value)

def predict_value(self, x, tree=None):

“”" Do a recursive search down the tree and make a prediction of the data sample by the

value of the leaf that we end up at “”"

if tree is None:

tree = self.root

If we have a value (i.e we’re at a leaf) => return value as the prediction

if tree.value is not None:

return tree.value

Choose the feature that we will test

feature_value = x[tree.feature_i]

Determine if we will follow left or right branch

branch = tree.false_branch

if isinstance(feature_value, int) or isinstance(feature_value, float):

if feature_value >= tree.threshold:

branch = tree.true_branch

elif feature_value == tree.threshold:

branch = tree.true_branch

Test subtree

return self.predict_value(x, branch)

def predict(self, X):

“”" Classify samples one by one and return the set of labels “”"

y_pred = [self.predict_value(sample) for sample in X]

return y_pred

def print_tree(self, tree=None, indent=" "):

“”" Recursively print the decision tree “”"

if not tree:

tree = self.root

If we’re at leaf => print the label

if tree.value is not None:

print (tree.value)

Go deeper down the tree

else:

Print test

print ("%s:%s? " % (tree.feature_i, tree.threshold))

Print the true scenario

print (“%sT->” % (indent), end=“”)

self.print_tree(tree.true_branch, indent + indent)

Print the false scenario

print (“%sF->” % (indent), end=“”)

self.print_tree(tree.false_branch, indent + indent)

class XGBoostRegressionTree(DecisionTree):

“”"

Regression tree for XGBoost

• Reference -

http://xgboost.readthedocs.io/en/latest/model.html

“”"

def _split(self, y):

“”" y contains y_true in left half of the middle column and

y_pred in the right half. Split and return the two matrices “”"

col = int(np.shape(y)[1]/2)

y, y_pred = y[:, :col], y[:, col:]

return y, y_pred

def _gain(self, y, y_pred):

nominator = np.power((y * self.loss.gradient(y, y_pred)).sum(), 2)

denominator = self.loss.hess(y, y_pred).sum()

return 0.5 * (nominator / denominator)

def _gain_by_taylor(self, y, y1, y2):

Split

y, y_pred = self._split(y)

y1, y1_pred = self._split(y1)

y2, y2_pred = self._split(y2)

true_gain = self._gain(y1, y1_pred)

false_gain = self._gain(y2, y2_pred)

gain = self._gain(y, y_pred)

return true_gain + false_gain - gain

def _approximate_update(self, y):

y split into y, y_pred

y, y_pred = self._split(y)

Newton’s Method

gradient = np.sum(y * self.loss.gradient(y, y_pred), axis=0)

hessian = np.sum(self.loss.hess(y, y_pred), axis=0)

update_approximation = gradient / hessian

return update_approximation

def fit(self, X, y):

self._impurity_calculation = self._gain_by_taylor

self._leaf_value_calculation = self._approximate_update

super(XGBoostRegressionTree, self).fit(X, y)

class RegressionTree(DecisionTree):

def _calculate_variance_reduction(self, y, y1, y2):

var_tot = calculate_variance(y)

var_1 = calculate_variance(y1)

var_2 = calculate_variance(y2)

frac_1 = len(y1) / len(y)

frac_2 = len(y2) / len(y)

Calculate the variance reduction

variance_reduction = var_tot - (frac_1 * var_1 + frac_2 * var_2)

return sum(variance_reduction)

def _mean_of_y(self, y):

value = np.mean(y, axis=0)

return value if len(value) > 1 else value[0]

def fit(self, X, y):

self._impurity_calculation = self._calculate_variance_reduction

self._leaf_value_calculation = self._mean_of_y

super(RegressionTree, self).fit(X, y)

class ClassificationTree(DecisionTree):

def _calculate_information_gain(self, y, y1, y2):

Calculate information gain

p = len(y1) / len(y)

entropy = calculate_entropy(y)

info_gain = entropy - p * \

calculate_entropy(y1) - (1 - p) * \

calculate_entropy(y2)

return info_gain

def _majority_vote(self, y):

most_common = None

max_count = 0

for label in np.unique(y):

Count number of occurences of samples with label

count = len(y[y == label])

if count > max_count:

most_common = label

max_count = count

return most_common

def fit(self, X, y):

self._impurity_calculation = self._calculate_information_gain

self._leaf_value_calculation = self._majority_vote
super(ClassificationTree, self).fit(X, y)

网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。

需要这份系统化学习资料的朋友，可以戳这里获取

一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！