基于zookeeper管理redis集群，LCMPS框架（二）

本文续上篇，介绍一个LCMPS框架（基于redis和zookeeper），LCMPS已经在项目中应用后期考虑进行开源。本文主要着重介绍LCMPS原理（简单英文），包含几个部分：

1.Overview

2.Concepts and Techniques

3.Case demo

4. Future plan
特别说明，LCMPS框架是由本人和一位同事（技术大牛）提出的，再次感谢他！

进入正题：

1.Overview

1.1.SDI concepts and techniques

•SOA

•EBUS

•Web Service

•RPC

•Message Queue

•Restful Service

•Socket programming

1.2.Whatis LCMPS?

LCMPS is designed for SDI, providing facilities for message publishing and subscribing. With LCMPS,distributable systems can share data in a looseness and scalable way. 

1.3.LCMPS Architecture

2.Concepts and Techniques

2.1.LCMPS concepts and techniques

• Redis  cluster

• Zookeeper  cluster

• Message Publish and Subscribe

• High availability and scalability

• High performance

• Workload balance

• Easy-to-use and Extensible API

2.1.1.Redis & Redis Cluster

• Redis is a high-performance memorycache platform.
    Clients can save data in the form of key-value into a redisnode called master.

    Clients can publish message to specific channels, and clients who subscribed thesechannels will get an immediate notice.

•Redis cluster consists of severaldecentralized redis nodes, allowing clients requests not a specific but poolednodes

    Data saved in the pre-allocated master following a Consistent Hash strategy.

    Node management and data migration between nodes.

    Published messages broadcast among all of the nodes.

2.1.2.Zookeeper cluster

•Provide distribution coordination service

    Redis Cluster management

    Configuration management

2.1.3.Message Publish/Subscribe

0. DA SUB query the redis master to subscribeMessage*(channel)

1. DA SUB register the message-notify callback

2. DA PUB query the redismaster to publish MessageX

3. DA PUB publish the message into the channel

4. Redis master B receive the broadcast message andnotify DA SUB

5. DA SUB get notified and invoke the callback toprocess theMessageX

2.1.4.Availability

• Redis cluster master collapse

• Zookeeper server collapse

2.1.5.Scalability

• Add a new redis master node

2.1.6.High performance

•Redis configuration: 3 masters, 3 slaves

•ChannelID: 01000000001

•Packetsize: 400bytes

•Subscriberedis: 99.1.15.117:6381 (master)

•Publishredis: 99.1.15.230:6381 (master)

•Publish/Subscribeclient: 99.0.64.7

•Throughput:7.6Mb/s

•TPS:2380TPS

2.1.7.Workload balance

•Classify Redis master intocategories.

•Zookeeper Redis master node statistics.

•Distribute PUB/SUB DA request allover the redis masters.

•Workload balance service is providedby Core API during the redis master node selection when DA pub or sub need topublish or subscribe channel message.

2.1.8.Core APIs for DA

• voidznodeWatch(znodeType,callback)

Ø DA register itself for Zookeeper Redis Master Node change notification.

• RCNode balancedRCNode(channel)

Ø DAget a balanced Redis Master Node for message Publish or Subscribe.

• void publish(channel,message,RCNode)

Ø Publishmessage into channel

• void publish(channel,message)

Ø Return Publish(channel, message,balancedRCNode(channel))

• Boolean subscribe(channel,RCNode)

Ø Subscribe channel

• Boolean subscribe(channel)

Ø Return subscribe(channel,balancedRCNode(channel))

• Map<channel,RCNode>rcNodeTakeover(da)

Ø DA detect an RC node failure, and try to register itself to another availableRedis Master

3.Case demo

3.1.EmulationProject (EMU, previous version)

3.2.AdoptLCMPS to EMU

4.Future plan

•DA high-availability design andimplementation

•Enhanced workload balance algorithm

•More Extensible API