Optimal BST
Assume:
-
We have a list of n items: a1, a2, ..., an
-
Key(ak) = ak
-
Probability of accessing item ak is known in advance and is P(ak)
-
The list is ordered by keys, a1 <= a2 <= ... an
How to produce the BST that has the least search cost given the access probability for each key?
Example:
-
list = a, b, c, d
-
P(a) = .1, P(b) = .2, P(c) = .3, P(d) = .4
![](https://i-blog.csdnimg.cn/blog_migrate/cff1b393fe9b12fea2f223534f1f1ef6.gif)
-
Optimal BST
![](https://i-blog.csdnimg.cn/blog_migrate/d0ca0e077eb03e3a15ffa425d9b79b0b.gif)
-
Average Cost = 1*0.3 + 2*0.4 + 2*0.2 + 3*0.1 =
1.8
Let A[j, k] = minimum average search time for a binary search tree with items aj <= aj+1 <= ... ak
![](https://i-blog.csdnimg.cn/blog_migrate/ff5173e60984cdfeeadb1e13f4d9e048.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/2189122f0fd49080b9b307142354f8ea.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/91c01f02aa919ac8305854beca88cf08.gif)
j\k | 0 | 1 | 2 | 3 | 4 | 5 |
1 | 0 | P(a1) | ||||
2 | 0 | 0 | P(a2) | A[2,4] | ||
3 | 0 | 0 | 0 | P(a3) | ||
4 | 0 | 0 | 0 | 0 | P(a4) | |
5 | 0 | 0 | 0 | 0 | 0 | P(a5) |
6 | 0 | 0 | 0 | 0 | 0 | 0 |
Time Complexity for finding Optimal BST Theta(n 3)
Dynamic Programming
"Dynamic programming algorithm stores the results for small subproblems and looks them up, rather than recomputing them, when it needs them later to solve larger subproblems"
Baase
"Dynamic programming algorithm is very useful approach to many optimization problems. ... Usually, we can find a tailor-made algorithm which is more efficient than the straight-forward algorithm based on dynamic programming"
T. C. Hu
Characterize the structure of an optimal solution
Recursively define the value of an optimal solution
Compute the value of an optimal solution in a bottom-up fashion
Construct an optimal solution from computed information
Let A be a 10 * 100 matrix,
-
B be a 100 * 5 matrix
C be a 5 * 50 matrix
-
A * B is a 10 * 5 matrix
B * C is a 100 * 50 matrix
Computing
-
A * B takes 10 * 100 * 5 = 5,000 multiplications
(A * B) * C takes 10 * 5 * 50 = 2,500 additional mult.
So (A * B) * C requires total of 7,500 multiplications
Computing
-
B * C takes 100 * 5 * 50 = 25,000 multiplications
A * (B * C) takes 10 * 100 * 50 = 50,000 additional mult.
So A * (B * C) requires total of 75,000 multiplications
But (A * B) * C = A * (B * C)
Given a chain <A1, A2, ..., An> matrices, where matrix Ak has dimension pk-1 * pk fully parenthesize the product A1* A2*... *An, in a way that minimizes the number of scalar multiplications
Optimal solution is of the form:
-
(A1* ... *Ak) * (Ak+1* ... *An)
not showing the parentheses in (A1* ... *Ak) or in (Ak+1* ... *An)
We must use the optimal parenthesization of A1* ... *Ak and the optimal parenthesization of Ak+1* ... *An
Thus the optimal solution to an instance of the matrix-chain multiplication problem contains within it optimal solutions to subproblem instances
That is it is recursive
Recall matrix Ak has dimension pk-1 * pk for k = 1,..., n
Let m[k, w] be the minimum the number of scalar multiplications needed to compute Ak* ... *Aw
If (Ak* ... *Az) * (Az+1* ... *Aw) is the optimal solution then
-
m[k, w] = m[k, z] + m[z+1, w] + pk-1 * pz * pk
since (Ak* ... *Az) is a pk-1 * pz matrix
and (Az+1* ... *Aw) is a pz * pw matrix
But what is z?
![](https://i-blog.csdnimg.cn/blog_migrate/1d89a2a7fc8d4a56d10b336bdc95f92c.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/b1095bb2275b7daa3cab32fe22e9277c.gif)
Example
Matrix dimension r pr A1 10*20 0 10 A2 20*3 1 20 A3 3*5 2 3 A4 5*30 3 5 4 30
1 | 2 | 3 | 4 | |
1 | 0 | 600 | 750 | 1950 |
2 | 0 | 300 | 2250 | |
3 | 0 | 450 | ||
4 | 0 |
![](https://i-blog.csdnimg.cn/blog_migrate/6df13e4b4232cadce4fd6e0627be8cab.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/e82f2591163b66425d503ebe64703139.gif)
Example
Matrix dimension r pr A1 10*20 0 10 A2 20*3 1 20 A3 3*5 2 3 A4 5*30 3 5 4 30
1 | 2 | 3 | 4 | |
1 | A1 | A1A2 | (A1A2)A3 | (A1A2)(A3A4) |
2 | A2 | A2A3 | A2(A3A4) | |
3 | A3 | A3A4 | ||
4 | A4 |
![](https://i-blog.csdnimg.cn/blog_migrate/28d1d12fc672067e1eb4adeaba1971e3.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/78e01d77fcb152d673213b290bd14bf8.gif)
We have a list of n items: a1, a2, ..., an
Probability of accessing item ak is P(ak)
Let A[j, k] = minimum average search time for a binary search tree with items aj <= aj+1 <= ... ak
![](https://i-blog.csdnimg.cn/blog_migrate/a92cf3686b385f294e65ccd791be845f.gif)
Let root[j,k] = p that gave the minimum value for A[j, k]
That is root[j,k] = root of OBST for items aj, a2, ..., ak
Let w[j,k] = P(aj) + P(aj+1) + ... + P(ak)
for k = 1 to n do
-
A[k, k] = P(ak)
A[k, k-1] = 0
root[k,k] = k
w[k, k] = P(ak)
A[n+1, n] = 0
for diagonal = 1 to n -1 do
-
for j = 1 to n - diagonal do
-
k = j + diagonal
w[j, k] = w[j, k - 1] + P(ak)
let p, j <= p <= k, be the value minimizes:
![](https://i-blog.csdnimg.cn/blog_migrate/7197d18b429b3c74a1e2cd9512065488.gif)
k 1 2 3 4 5 6 ak a b c d e f P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25
root
1 1 1 1 1 3 0 2 3 3 3 5 0 0 3 3 3 5 0 0 0 4 5 6 0 0 0 0 5 6 0 0 0 0 0 6
A
0 0.4 0.5 0.85 1 1.35 2.1 0 0 0.05 0.25 0.35 0.6 1.2 0 0 0 0.15 0.25 0.5 1.05 0 0 0 0 0.05 0.2 0.6 0 0 0 0 0 0.1 0.45 0 0 0 0 0 0 0.25 0 0 0 0 0 0 0
![](https://i-blog.csdnimg.cn/blog_migrate/b5d3fbb622d2b4cc2ebc5adef141e8ba.gif)
P(ak)'s = (0.15 0.025 .05 .025 .05 .125 .025 .075 0.075 .05 .15 .075 .05 .025 .05)
Root
1 1 1 1 1 3 3 6 6 6 6 6 6 6 6 0 2 3 3 3 5 6 6 6 6 6 9 9 9 11 0 0 3 3 3 5 6 6 6 6 9 9 9 9 11 0 0 0 4 5 6 6 6 6 6 9 9 9 9 11 0 0 0 0 5 6 6 6 6 8 9 9 9 11 11 0 0 0 0 0 6 6 6 8 8 9 9 11 11 11 0 0 0 0 0 0 7 8 8 9 9 11 11 11 11 0 0 0 0 0 0 0 8 8 9 9 11 11 11 11 0 0 0 0 0 0 0 0 9 9 11 11 11 11 11 0 0 0 0 0 0 0 0 0 10 11 11 11 11 11 0 0 0 0 0 0 0 0 0 0 11 11 11 11 11 0 0 0 0 0 0 0 0 0 0 0 12 12 12 13 0 0 0 0 0 0 0 0 0 0 0 0 13 13 13 0 0 0 0 0 0 0 0 0 0 0 0 0 14 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15
A[1,15] = 2.925
for diagonal = 1 to n -1 do
-
for j = 1 to n - diagonal do
-
k = j + diagonal
w[j, k] = w[j, k - 1] + P(ak)
let p,
root[j,k-1] <= p <= root[j+1,k], be the value minimizes:
![](https://i-blog.csdnimg.cn/blog_migrate/a841aea8ba2e59eca2fad424cd2e1c0a.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/0161c65d674e300431e5542b01c1d03d.gif)
Let H(i, j) be a real number for 1 <= i < j <= n
Let c(i, j) be defined by:
-
c(i, i) = 0
c(i, j) = H(i, j) +
![](https://i-blog.csdnimg.cn/blog_migrate/d075a90b8e4fa4a777d6574f1e498f2d.gif)
H(i, j) is monotone with respect to set inclusion of intervals if
-
H(j, k) <= H(x, y) if j <= x < y <= k
H(i, j) satisfies the quadrangle inequality (QI) if
-
H(j, k) + H(x, y) <= H(x, k) + H(j, y) if j <= x < k <= y
Theorem. If H(i, j) satisfies QI and is monotone then c(i, j) can be computed in time O(n 2)
Lemma. If H(i, j) satisfies QI and is monotone then c(i, j) satisfies QI
- K(i, j) <= K(i, j+ 1) <= K(i+1, j+1)
Splay vs. OBST
Recall in OBST we have:
-
We have a list of n items: a1, a2, ..., an and leaves b0, b1, ..., bn
Probability of accessing item ak is P(ak) =
Alphak
Let
Betak be the probability of accessing a key that is between ak and ak+1, that is leaf bk
In splay tree we have Betak = 0, and Alphak = q(i)/m
Theorem 9: Let Popt be the be the weighted path length of optimum BST. We have:
![](https://i-blog.csdnimg.cn/blog_migrate/490442098c67746f3ae9c5a18b6ffee9.gif)
Thus we have:
![](https://i-blog.csdnimg.cn/blog_migrate/5b829da0754c1dae4269f201319c15eb.gif)
Now:
![](https://i-blog.csdnimg.cn/blog_migrate/a29c29f4c39ae614151e65b2cf851363.gif)
So m*H =
![](https://i-blog.csdnimg.cn/blog_migrate/e322192c682d64017f49eb1cb19afe40.gif)
Thus: m*Popt <= m*H + 1 =
![](https://i-blog.csdnimg.cn/blog_migrate/348848a95b5df9739f52ac2ef647e6de.gif)
Nearly Optimal BST
We have a list of n items: a1, a2, ..., an
Probability of accessing item ak is P(ak)
k 1 2 3 4 5 6
ak a b c d e f
P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25
![](https://i-blog.csdnimg.cn/blog_migrate/cfa4f79ee60697b063ab35e9ce357967.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/def1b5c09635579066a90f199b8ac346.gif)
Average Access Cost = 2.2
ak a b c d e f
P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25
![](https://i-blog.csdnimg.cn/blog_migrate/05b2752b7e045bc7769021de7475c374.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/32e7180764ed6d14f412cb0be8cfec51.gif)
Average Access Cost = 2.1
Optimal Alphabetic Tree[1]
An alphabetic tree is a binary search tree in which all data is in the leaves. Internal nodes are used in search for the data
![](https://i-blog.csdnimg.cn/blog_migrate/1d0065c28372654f17de40ec2739de95.gif)
Let V1, V2,... Vn be the order of the leaves
Let wk be the weight, or frequency of access, of leaf Vk
Combining Vk and Vp, denote their parent node by Vkp and it weight wkp = wk+ wp
All leaves are square nodes, all parents are round nodes
Two nodes are a compatible pair if they are adjacent or if all nodes between them are round nodes
The weight of a pair is the weight of the parent of the two nodes
A pair with minimum weight over all pairs is a minimum pair
Minimum compatible pair is the compatible pair with the least weight over all compatible pairs
-
Ties are broken by taking the pair with the left most left node
If two compatible pairs have the same left node and the same weight, then pick the pair with the leftmost right node
1. Construction
-
Find the minimum compatible pair
Replace the left node of the pair by the pair's parent
Remove right node of the pair
Repeat n-1 times
Call the resultant tree T'
2. Level Assignment
-
Determine the level number Lk of every leaf Vk in T'
-
We have the level numbers L1, L2,... Ln of all leaves
Find the leftmost maximum level number, say Lk = q
Then Lk+1 = q
Replace Lk and Lk+1 with a parent node with level q - 1
Repeat n - 1 times
Theorem. The Hu-Tucker algorithm can be implemented to produce the optimal alphabetic tree with N leaves in O(Nlg(N)) time and O(N) space
We have a list of n items: a1, a2, ..., an and leaves b0, b1, ..., bn
Probability of accessing item ak is known in advance and is P(ak) = Alphak
Let Betak be the probability of accessing a key that is between ak and ak+1, that is leaf bk
The list is ordered by keys, b0< a1 <b1< a2 < ... < an < bn
![](https://i-blog.csdnimg.cn/blog_migrate/22c3676750c94b0466080f122c488a69.gif)
( Beta0 , Alpha1 , Beta1 , Alpha2, Beta2,... , Alphan, Betan) is the access distribution
Let
![](https://i-blog.csdnimg.cn/blog_migrate/135220428b6a084343ba9d510352745c.gif)
Let L(ak) be the level of the node ak
Let
![](https://i-blog.csdnimg.cn/blog_migrate/9a5993293523bfc93adab45a36a0ed24.gif)
P is the weighted path length of a tree
Let ( Gamma1 , Gamma2,... , Gamman) be a discrete probability distribution, i.e.
Gammak >= 0 and SigmaGammak =1
H( Gamma1 , Gamma2,... , Gamman) =
![](https://i-blog.csdnimg.cn/blog_migrate/98e666d4a905b6f99c2778c046456b5e.gif)
is the entropy of the distribution. ( 0*log 0 = 0)
Let TBB be the tree resulting from the nearly optimal BST algorithm 1
Theorem 7 [2]: Let L(ak) be the depth of node ak and let L(bk) be the depth of leaf bk in tree TBB Then
-
bk <= floor (log 1/bk) + 2
ak <= floor (log 1/ak)
Theorem 8 [3]: Let PBB be the weighted path length of the tree TBB. Then
![](https://i-blog.csdnimg.cn/blog_migrate/2f426caa4aa7b95e0dc9b3e534d1562b.gif)
Theorem 9 [4]: Let PBB be the weighted path length of tree TBB. Let Popt be the be the weighted path length of optimum BST. We have:
![](https://i-blog.csdnimg.cn/blog_migrate/ccb5eb309bf0b8b25131b2fb63aff4f7.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/08b1b879321c39b2721e0c257c4fa8d3.gif)
This gives us
![](https://i-blog.csdnimg.cn/blog_migrate/7c8dcebeef43d9c47845012c34eea997.gif)
In English, the probability of occurrence of the i-th most frequent word is approximately [5]
![](https://i-blog.csdnimg.cn/blog_migrate/9e5ee91f0fa69c8a8461e8e07998b550.gif)
This yields
H( Alpha1 , Alpha2, ... ) = 10.2
The weighted path length of an optimum binary search tree for all English words is no larger than 11.2.
We have a list of n items: a1, a2, ..., an
Probability of accessing item ak is P(ak)
k 1 2 3 4 5 6
ak a b c d e f
P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25
![](https://i-blog.csdnimg.cn/blog_migrate/6983f1a27fe5d4641657cd55a506643a.gif)
Problem:
k 1 2 3
ak a b c
P(ak)'s = 0.45 0.1 0.45
![](https://i-blog.csdnimg.cn/blog_migrate/81da684e89e9c4e075455aab420dc362.gif)
Algorithm for Nearly Optimal Lexicographic Tree[6]
Given the ordered set {a} of names, such that a1 <= a2 <= ... an , and two parameters, F and N0
(1) If N <= N0 , use dynamic programming.
(2) If N> N0, let W[k, l] be the weight of the subtree with frequencies
-
Betak ,
Alphak+1 ,
Betak+1 , ... ,
Alphal,
Betal
(3) Find an index, max, such that Alphamax = maximum Alphai, where ai is in {AF}.
(4) If in the set {AF} there is at least one name preceding or equal to AC with associated frequency Alphamax, let p be the index such that ap with Alphap = Alphamax, is lexicographically closest to AC.
-
If there is no such p, let {AQ} be the null set and go to Step 6.
(5) If ap is the first member of {AF} and Alphap-1 > Alphap, form the set {AQ} = {ap-1 , ap-2, ., au}, where Alphap-j-1 > Alphap-j , j = 0, ... ,p-u-1 and Alphau-1 <= Alphau or u-p = floor(lg N);
-
if ap is not the first member of {AF}, let {AQ} be the null set.
(6) If in the set {AF} there is at least one name following or equal to AC with associated frequency Alphamax let r be the index such that ar with Alphar = Alphamax is lexicographically closest to AC;
-
if there is no such r, let {AS} be the null set and go to Step 8.
(7) If ar is the last member of {AF} and Alphar< Alphar+1, form the set {AS}= {ar+1, ar+2,...,av}, where Alphar+j< Alphar+j+1, j=0, 1, ... ,v-r-1, and Alphav>= Alphav+1or v-r = floor(lg N).
-
If ar is not the last member of {AF}, let {AS} be the null set.
(9) Go to Step 1 and repeat the algorithm for the subtrees a1, a2, ... aroot-1 and aroot+1, ... aN, where N is root-I and N-root for the two cases.
if Beta/ Alpha is small (<= 3) than use N0= 15
if Beta/ Alpha is large (> 3) than use N0= 25 or 30
if Beta/ Alpha is large (> 2) than use F = 6
if Beta/ Alpha ~ 1 than use F = 4
if Beta/ Alpha < 1 than use F = 4
The tree takes O(Nlog(N)) to construct
Average search time for NOBST is within 2% of the average search time of the OBST
N0= 15, F = 5
N | Alpha freq | Beta freq | OBST | NOBST |
5 | 19,846 | 982,497 | 3.4114 | 3.4114 |
15 | 42,653 | 959,690 | 4.2864 | 4.2864 |
25 | 60,087 | 942,256 | 5.0638 | 5.1033 |
50 | 92,117 | 910,226 | 6.0483 | 6.1461 |
100 | 138,975 | 863,368 | 7.0007 | 7.0437 |
150 | 173,157 | 829,186 | 7.4885 | 7.5503 |
200 | 200,412 | 801,931 | 7.8795 | |
500 | 305,266 | 697,077 | 8.9606 | |
1000 | 401,288 | 601,055 | 9.6490 | |
3000 | 561,956 | 440,387 | 10.6220 | |
6000 | 655,538 | 346,805 | 11.1177 | |
12000 | 740,022 | 262,321 | 11.1592 | |