Cache Python MySQL Result using Redis
We’re often using MySQL so much, like read data from DB, calculate something, write data with index. And I bet You know, that process is also using your i/o machine. I prefer my CPU usage 100% than my i/o 50%.
Redis is live in memory, so it won’t do anything like read / write to disk that make your i/o process high (untill your max memory, if it’s not enough, redis create virtual memory, CMIIW). I plan to make my application faster and faster with caching using redis, and I will improve my news web crawler to going faster with redis as cache.
The code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
|
#!/usr/bin/python
# Author : Fajri Abdillah a.k.a clasense4
# Twitter : @clasense4
# mail : clasense4@gmail.com
# import modules used here -- sys is a very standard one
import
sys
import
db_base_local
# ==> Your MySQL Connection
import
redis
# ==> Make sure to install this library using pip install redis
from
datetime
import
datetime
import
time
import
cPickle
import
hashlib
# START TIME
startTime
=
datetime.now()
# CURSOR DB
CURSOR
=
db_base_local.conn.cursor()
# Redis Object
R_SERVER
=
redis.Redis(
"localhost"
)
sql
=
"select * from news_crawler_rss"
# Or Any SQL script
# Gather our code in a main() function
def
cache_redis(sql, TTL
=
36
):
# INPUT 1 : SQL query
# INPUT 2 : Time To Life
# OUTPUT : Array of result
# Create a hash key
hash
=
hashlib.sha224(sql).hexdigest()
key
=
"sql_cache:"
+
hash
print
"Created Key\t : %s"
%
key
# Check if data is in cache.
if
(R_SERVER.get(key)):
print
"This was return from redis"
return
cPickle.loads(R_SERVER.get(key))
else
:
# Do MySQL query
CURSOR.execute(sql)
data
=
CURSOR.fetchall()
# Put data into cache for 1 hour
R_SERVER.
set
(key, cPickle.dumps(data) )
R_SERVER.expire(key, TTL);
print
"Set data redis and return the data"
return
cPickle.loads(R_SERVER.get(key))
# Standard boilerplate to call the main() function to begin
# the program.
if
__name__
=
=
'__main__'
:
cache_redis(sql)
|
Explanation
- Check Redis Key, if exists, just return it
- If not exists, do MySQL query, and set the key, then return it
Why I use serialization with cPickle? because Redis just store the string with SET Command, so, our MySQL result must serialize to string, and cPickle is faster than pickle or marshal (cmiiw).
Conclusion
This simple post is just to show how to use cache python mysql result using redis. And in my opinion, redis just awesome
Better code view