I'm currently playing with the Typeahead problem on Talentbuddy. There are already 2 questions on this subject (Typeahead Talent Buddy and Typeahead autocomplete functionality challenge), but none of these use a trie which is why I created a new one.
Problem Statement
Given a big list of user names and a list of strings representing queries Your task is to:
- Write a function that prints to the standard output (stdout) for each query the user name that matches the query
- If there are multiple user names matching the query please select the one that is the smallest lexicographically
- All string matches must be case insensitive
- If no match is found for a given query please print "-1"
Note that your function will receive the following arguments: usernames
Which is an array of strings representing the user names queries
Which is an array of strings representing the queries described above
Data constraints
The length of the array above will not exceed 100,000 entries
Each name or query string will not exceed 30 characters
Efficiency constraints
- Your function is expected to print the requested result and return in less than 2 seconds
Example
names: ["james", "jBlank"] queries: ["j", "jm", "jbl", "JB"] Output: james -1 jBlank jBlank"
I quickly whipped up a brute force solution which was far too slow (about 100 seconds on my laptop for one of the medium test data sets). Then I started researching and came to the conclusion that a trie is probably the best data structure for this. I first went with a generic trie which improved the speed by about 100x. This got me a bit further in the tests, but they try with a few bigger data sets after that and with the last one the algorithm is still too slow.
I've optimized the trie specifically for the problem, then profiled and finetuned the implementation, which got me another 100x speed improvement. However, my implementation still barely makes the 2 second time limit on my machine (1.1 seconds on average with cpython) and doesn't make it on the Talentbuddy server.
Current Implementation:
My algorithm works by first creating a prefix tree (trie) with the usernames. The trie is build from dicts, where a dict entry for a letter references it's corresponding trie node (which is a dict itself). Each node also stores the lexicographic smallest word matching the prefix used to get to the node in it's dict with a special key ("_word").
Creating the trie works by traversing the trie letter by letter for each word and overwriting the "_word" mapping in each encountered node with the current word. Because the word list is sorted in reverse order, this ensures each node will store the lexicographic smallest word with the prefix used to arrive there.
The trie is queried by traversing it using the letters from the query. The node we arrive at the last letter stores the lexicographic smallest username matching this prefix.
class Trie(object):
""" Simple trie implementation for autocompletion.
Build this trie once by instanciating it with a list of words.
You can then get the smalles lexicographic match by calling the by_prefix
method.
Note: While storing text unaltered, the trie ignores case.
"""
_word = "_word"
def __init__(self, words):
self.root = {}
# presorting ensures every time we traverse a node the current word will
# be lexicographically smaller, thus we can always replace it.
words.sort(key=str.lower, reverse=True)
for word in words:
node = self.root
for char in word.lower():
node = node.setdefault(char, {})
node[self._word] = word
def by_prefix(self, prefix):
"""Return lexicographically smallest word that starts with a given
prefix.
"""
curr = self.root
for char in prefix.lower():
if char in curr:
curr = curr[char]
else:
return "-1"
else:
return curr[self._word]
def typeahead(usernames, queries):
"""Given a list of users and queries,
this function prints all valid users for these queries.
Args:
usernames: list of strings representing users.
queries: list of strings representing (incomplete) input
"""
users = Trie(usernames)
print "\n".join(users.by_prefix(q) for q in queries)
Complexity
The worst case complexity for a lookup in the trie now is O(n) where n is the length of a query (which can't be larger than 30).
Creating the trie is a bit worse, with O(n*m) where n is the size of the username list and m the average length of a username.
Optimization
I have used the line profiler to find out where most time is spent. I have modified the typeahead function to be able to pinpoint the bottlenecks better:
Total time: 1.01042 s File: typeahead.py Function: __init__ at line 13 Line # Hits Time Per Hit % Time Line Contents ============================================================== 13 @profile 14 def __init__(self, words): 15 1 1 1.0 0.0 self.root = {} 16 # presorting ensures every time we traverse a node the current word will 17 # be lexicographically smaller, thus we can always replace it. 18 1 38859 38859.0 3.8 words.sort(key=str.lower, reverse=True) 19 50001 22553 0.5 2.2 for word in words: 20 50000 20530 0.4 2.0 node = self.root 21 615946 249450 0.4 24.7 for char in word.lower(): 22 565946 419829 0.7 41.6 node = node.setdefault(char, {}) 23 565946 259193 0.5 25.7 node[self._word] = word Total time: 0.402196 s File: typeahead.py Function: by_prefix at line 25 Line # Hits Time Per Hit % Time Line Contents ============================================================== 25 @profile 26 def by_prefix(self, prefix): 27 """Return lexicographically smallest word that starts with a given 28 prefix. 29 """ 30 50000 19609 0.4 4.9 curr = self.root 31 332685 130631 0.4 32.5 for char in prefix.lower(): 32 288221 114342 0.4 28.4 if char in curr: 33 282685 112530 0.4 28.0 curr = curr[char] 34 else: 35 5536 1950 0.4 0.5 return "-1" 36 else: 37 44464 23134 0.5 5.8 return curr[self._word] Total time: 3.46204 s File: typeahead.py Function: typeahead at line 39 Line # Hits Time Per Hit % Time Line Contents ============================================================== 39 @profile 40 def typeahead(usernames, queries): 41 """Given a list of users and queries, 42 this function prints all valid users for these queries. 43 44 Args: 45 usernames: list of strings representing users. 46 queries: list of strings representing (incomplete) input 47 """ 48 1 1745292 1745292.0 50.4 users = Trie(usernames) 49 1 3 3.0 0.0 results = (users.by_prefix(q) for q in queries) 50 1 844213 844213.0 24.4 output = "\n".join(results) 51 1 872537 872537.0 25.2 print output
Possible Bottlenecks
- Dictionary creation takes almost 40% of the cost of building the trie. Is there any other data structure in Python suited for fast lookup? In C I would use an array of pointers, but there is no such thing in Python and from looking at the alternatives I gather that classes or named tuples also use a dict to perform member dereferencing. Would using a list and indexing it by (letter - ord('a')) be faster?
- When performing the lookups in the trie most of the cost (almost 60%) comes from dictionary lookups. Solving Bottleneck 1 would most likely also solve this one.
- 25% of overall time is spent printing the output to the terminal. Unfortunately there is not much we can do about that. I already "optimized" this by doing only 1 call to print, because it blocks on every newline... This is somewhat of a problem with the format talentbuddy uses for output validation. However I hope they at least don't use a terminal on the validation machine but redirect stdout.
Do you have any idea how I can further speed this up? I expect it needs to run at least 2x faster to pass on the Talentbuddy machine.
扫一扫
1万+
到【灌水乐园】发言
