一致性哈希算法

普通的哈希算法采用简单取模的方式,将缓存服务器进行散列,通常情况下是没有问题的,但是当缓存服务器的个数发生变动时,将会产生较大的影响

如上图所示,之前有4台缓存服务器,当增加1台缓存服务器之后,除数的变化(4 -> 5)导致求模结果变化,所有缓存查询均未命中

即缓存服务器的个数发生变化时,在一段时间内(缓存重建完毕之前),会有大量缓存查询未命中,导致这段时间内的服务整体性能下降特别严重

一致性哈希算法能有效降低服务器个数变化对整体缓存的影响,基本实现原理是将Hash函数的值域空间组织成一个圆环,将服务器节点进行哈希,并将哈希结果映射到圆环上,当有一个写入缓存的请求到来时,使用相同的Hash函数,计算Key的哈希值在圆环上对应的位置,按顺时针方向,将请求定位至离其最近的服务器节点

如下图所见,当增加一台缓存服务器Server5后,Server4和Server5之间的点将被定位至Server5,Server5和Server之间的点依然定位至Server,并且对Server2,Server3和Server4没影响,比起简单的求模哈希,未命中的缓存查询少了很多,整体服务性能不会下降过大

当然在实际使用过程中会在圆环上添加很多虚拟缓存服务器节点,以便缓存分布更加均匀

介绍完原理,我们再来看一下具体实现,以Memcached-java-client为例

如果我们想使用一致性哈希算法,只需要添加pool.setHashingAlg(SockIOPool.CONSISTENT_HASH);这行代码即可

  1. import com.danga.MemCached.MemCachedClient;
  2. import com.danga.MemCached.SockIOPool;
  3. public class Test {
  4. public static void main(String[] args) {
  5. MemCachedClient client = new MemCachedClient();
  6. String[] servers = { "192.168.52.129:9999",
  7. "192.168.52.131:9999"};
  8. Integer[] weights = { 1, 1};
  9. SockIOPool pool = SockIOPool.getInstance();
  10. pool.setServers(servers);
  11. pool.setWeights(weights);
  12. pool.setInitConn( 5);
  13. pool.setMinConn( 5);
  14. pool.setMaxConn( 250);
  15. pool.setMaxIdle( 1000 * 60 * 60 * 6);
  16. pool.setMaintSleep( 30);
  17. pool.setNagle( false);
  18. pool.setSocketTO( 3000);
  19. pool.setSocketConnectTO( 0);
  20. pool.setHashingAlg(SockIOPool.CONSISTENT_HASH);
  21. pool.initialize();
  22. client.set( "test", "This is a test String");
  23. String test = (String) client.get( "test");
  24. System.out.println(test);
  25. }
  26. }

来看下实际效果

  1. sean@ubuntu1:~$ telnet 192.168.52.131 9999
  2. Trying 192.168.52.131...
  3. Connected to 192.168.52.131.
  4. Escape character is '^]'.
  5. get test
  6. END
  7. sean1@ubuntu2:~$ telnet 192.168 .52 .129 9999
  8. Trying 192.168 .52 .129...
  9. Connected to 192.168 .52 .129.
  10. Escape character is '^]'.
  11. get test
  12. VALUE test 32 21
  13. This is a test String
  14. END

先从SockIOPool的初始化开始

  1. public void initialize() {
  2. ......
  3. if ( this.hashingAlg == 3)
  4. populateConsistentBuckets();
  5. else
  6. populateBuckets();
  7. ......
  8. }

构建一致性哈希算法中的整个圆环,当然从具体实现上来看只是构建虚拟节点的集合

  1. private void populateConsistentBuckets(){
  2. this.consistentBuckets = new TreeMap();
  3. MessageDigest localMessageDigest = (MessageDigest)MD5.get();
  4. // 获得总权重
  5. // 如果指定了每个服务器的权重,则其和值为总权重
  6. // 否则每个服务器权重为1,总权重为服务器个数
  7. if (( this.totalWeight.intValue() <= 0) && ( this.weights != null))
  8. for (i = 0; i < this.weights.length; ++i){
  9. SchoonerSockIOPool localSchoonerSockIOPool = this;
  10. (localSchoonerSockIOPool.totalWeight = Integer.valueOf(localSchoonerSockIOPool.totalWeight.intValue()
  11. + (( this.weights[i] == null) ? 1 : this.weights[i].intValue())));
  12. }
  13. else if ( this.weights == null)
  14. this.totalWeight = Integer.valueOf( this.servers.length);
  15. // 循环遍历每一个服务器以便创建其虚拟节点
  16. for ( int i = 0; i < this.servers.length; ++i){
  17. int j = 1;
  18. if (( this.weights != null) && ( this.weights[i] != null))
  19. j = this.weights[i].intValue();
  20. // 每个服务器的虚拟节点个数需参照该服务器的权重
  21. double d = Math.floor( 40 * this.servers.length * j / this.totalWeight.intValue());
  22. long l = 0L;
  23. // 循环构建每一个节点
  24. while (l < d){
  25. byte[] arrayOfByte = localMessageDigest.digest( this.servers[i] + "-" + l.getBytes());
  26. for ( int k = 0; k < 4; ++k){
  27. Long localLong = Long.valueOf((arrayOfByte[( 3 + k * 4)] & 0xFF) << 24
  28. | (arrayOfByte[( 2 + k * 4)] & 0xFF) << 16
  29. | (arrayOfByte[( 1 + k * 4)] & 0xFF) << 8
  30. | arrayOfByte[( 0 + k * 4)] & 0xFF);
  31. // 将每个虚拟节点添加到圆环中
  32. this.consistentBuckets.put(localLong, this.servers[i]);
  33. }
  34. l += 1L;
  35. }
  36. Object localObject;
  37. // 构建socket工厂类
  38. if ( this.authInfo != null)
  39. localObject = new AuthSchoonerSockIOFactory( this.servers[i], this.isTcp, this.bufferSize,
  40. this.socketTO, this.socketConnectTO, this.nagle, this.authInfo);
  41. else
  42. localObject = new SchoonerSockIOFactory( this.servers[i], this.isTcp, this.bufferSize,
  43. this.socketTO, this.socketConnectTO, this.nagle);
  44. // 使用socket工厂类创建连接池
  45. GenericObjectPool localGenericObjectPool = new GenericObjectPool((PoolableObjectFactory)localObject,
  46. this.maxConn, 1, this.maxIdle, this.maxConn);
  47. ((SchoonerSockIOFactory)localObject).setSockets(localGenericObjectPool);
  48. // 每个服务器都有自己的连接池
  49. this.socketPool.put( this.servers[i], localGenericObjectPool);
  50. }
  51. }

MemcachedClient的初始化方法,通过该方法可确定Client的具体实现类为AscIIUDPClient

  1. public MemCachedClient() {
  2. this( null, true, false);
  3. }
  4. public MemCachedClient(String paramString, boolean paramBoolean1,
  5. boolean paramBoolean2) {
  6. this.BLAND_DATA_SIZE = " ".getBytes();
  7. if (paramBoolean2)
  8. this.client = new BinaryClient(paramString);
  9. else
  10. this.client = new AscIIUDPClient(paramString);
  11. }

当发送一个添加请求时,本质还是通过调用set方法实现的

  1. public boolean add(String paramString, Object paramObject) {
  2. return set( "add", paramString, paramObject, null, null,
  3. Long.valueOf( 0L));
  4. }
  5. // paramInteger的值为null
  6. private boolean set(String paramString1, String paramString2,
  7. Object paramObject, Date paramDate, Integer paramInteger,
  8. Long paramLong) {
  9. ......
  10. SchoonerSockIO localSchoonerSockIO = this.pool.getSock(paramString2,
  11. paramInteger);
  12. ......
  13. }

服务器的查找过程如下

  1. public final SchoonerSockIO getSock(String paramString, Integer paramInteger) {
  2. ......
  3. // 计算Key的哈希值,并根据该哈希值得到对应的服务器节点哈希值
  4. long l = getBucket(paramString, paramInteger);
  5. // 根据服务器节点哈希值得到对应的服务器
  6. String str1 = ( this.hashingAlg == 3) ? (String) this.consistentBuckets
  7. .get(Long.valueOf(l)) : (String) this.buckets.get(( int) l);
  8. while (!(((Set) localObject).isEmpty())) {
  9. // 从服务器连接池中获取到特定服务器的连接
  10. SchoonerSockIO localSchoonerSockIO = getConnection(str1);
  11. ......
  12. }

首选根据Key值计算出其哈希值(getHash),然后根据得到的哈希值确定其在圆环上对应的服务器节点(findPointFor)

  1. // paramInteger的值为null
  2. private final long getBucket(String paramString, Integer paramInteger) {
  3. long l1 = getHash(paramString, paramInteger);
  4. if ( this.hashingAlg == 3)
  5. return findPointFor(Long.valueOf(l1)).longValue();
  6. long l2 = l1 % this.buckets.size();
  7. if (l2 < 0L)
  8. l2 *= - 1L;
  9. return l2;
  10. }

Key的哈希值计算过程如下,和populateConsistentBuckets方法中用来生成服务器虚拟节点哈希值的算法是一样的

  1. // paramInteger的值为null
  2. private final long getHash(String paramString, Integer paramInteger) {
  3. if (paramInteger != null) {
  4. if ( this.hashingAlg == 3)
  5. return (paramInteger.longValue() & 0xFFFFFFFF);
  6. return paramInteger.longValue();
  7. }
  8. switch ( this.hashingAlg) {
  9. case 0:
  10. return paramString.hashCode();
  11. case 1:
  12. return origCompatHashingAlg(paramString);
  13. case 2:
  14. return newCompatHashingAlg(paramString);
  15. case 3:
  16. return md5HashingAlg(paramString);
  17. }
  18. this.hashingAlg = 0;
  19. return paramString.hashCode();
  20. }
  21. private static long md5HashingAlg(String paramString) {
  22. MessageDigest localMessageDigest = (MessageDigest) MD5.get();
  23. localMessageDigest.reset();
  24. localMessageDigest.update(paramString.getBytes());
  25. byte[] arrayOfByte = localMessageDigest.digest();
  26. long l = (arrayOfByte[ 3] & 0xFF) << 24 | (arrayOfByte[ 2] & 0xFF) << 16
  27. | (arrayOfByte[ 1] & 0xFF) << 8 | arrayOfByte[ 0] & 0xFF;
  28. return l;
  29. }

在圆环上查找Key的哈希值对应的服务器节点哈希值

参照populateConsistentBuckets中的代码,所有虚拟节点被存放在一个TreeMap中,所以这里可以使用tailMap方法获得大于等于Key哈希值的子树,然后获取该树中最小值即可

  1. private final Long findPointFor(Long paramLong) {
  2. SortedMap localSortedMap = this.consistentBuckets.tailMap(paramLong);
  3. return ((localSortedMap.isEmpty()) ? (Long) this.consistentBuckets
  4. .firstKey() : (Long) localSortedMap.firstKey());
  5. }
版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/a19881029/article/details/52766698
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值