Optimal binning algorithm using dynamic programming

最新推荐文章于 2022-11-12 17:18:24 发布

US_579

最新推荐文章于 2022-11-12 17:18:24 发布

阅读量758

点赞数

分类专栏： python 算法文章标签： lab binning Optimal binning algorithm using dyn 分箱

本文链接：https://blog.csdn.net/US_579/article/details/88428294

版权

python 同时被 2 个专栏收录

9 篇文章 0 订阅

订阅专栏

算法

1 篇文章 0 订阅

订阅专栏

COMP9318-19T1

LAB2

Optimal binning algorithm using dynamic programming

Error Sum of Squares (SSE)

SSE is the sum of the squared differences between each observation and its group’s mean. It can be used as a measure of variation within a cluster. If all cases within a cluster are identical the SSE would then be equal to 0.

The formula for SSE is:

$SSE = \sum_{i=1}^{n}\left ( x_{i} \right -\bar{x})^2$

Where n is the number of observations xi is the value of the ith observation and 0 is the mean of all the observations. This can also be rearranged to be written as seen in J.H. Ward’s paper.

Recurisive method will compute many times, although its space complicity is lower than dynamic programming.

In this case we apply dynamic programming, like Fibonacci sequence , we use privious consequence to compute the next by storing it in the mermory.

this problem also implement this strategy

the first row ,represent from that block divide into one bin

[ ] [ ] [ ] [3, 1, 18, 11, 13, 17] -1

[ ] [ ] [ ] [1, 18, 11, 13, 17] -1

[ ] [ ] [ ] [18, 11, 13, 17] -1

[ ] [ ] [ ] [11, 13, 17] 18.666

[ ] [ ] [ ] [13, 17] 8.0

[ ] [ ] [ ] [17] 0

the second row ,represent from that block divide into two bin

[ ] [ ] [[3, 1, 18, 11, 13, 17]] -1

[ ] [ ] [[1, 18, 11, 13, 17]] -1

[ ] [ ] [[18, 11, 13, 17]] 18.666

[ ] [ ] [[11, 13, 17]] 2

[ ] [ ] [[13, 17]] 0

[ ] [ ] [[17]] -1

etc…

Divding the problem into sub-problem , store the first row because it is the best optimal solution [11, 13, 17]:18.666, [13, 17]:8.0,[17]:0, after that when we are going to the second row,it will divide into two bins from the first number in the list, so[18, 11, 13, 17] can be split to prefix and suffix,which are [18] and [11,13,17], (this is only the first situation,we still can split into [[18,13],[13,17]] [[18,11,13],[17]] and the rest will apply the same thinking)as we already know what the cost of[11,13,17] from the first row’s data which is 18.666(this is optimal)and the cost of [18] which is 0 , we are able to calculate the optimal cost that dividing [18, 11, 13, 17]into two bins is 0 + 18.999 == 18.999, so in the second row , the optimal cost of dividing [18,11,13,17] into two bins is 18.999 this is the core thinking of the dynamic programming.