非jvm语言如何定制kafka api

     需求背景: 使用node.js的前端同学需要在程序里动态创建kafka topic。毫无疑问肯定先从kafka官网或者github找，可是找到的都是基于kafka服务端开启auto.create.topics.enable然后模拟request请求来达到，这种方式的局限是无法设置自己需要的分区数和副本数(只能使用kafka服务端配置文件的固定参数)。还有一种方式可以通过跟shell交互模拟命令行创建topic，但是这种方式需要jar包和jvm环境，这对于使用node.js的同学肯定是不希望采用的。综上，我们必须使用node.js去开发一个创建topic的api。
    首先我们分析kafka源码看看创建topic的过程做了哪些事情(以下是基于kafka0.8.2.1的源码进行分析)。我们一般知道的都是通过命令行来创建topic，那就从kafka-topics.sh这个文件切入，从这个shell可以看到最终是使用了kafka.admin.TopicCommand这个类来处理，对应的是kafka源码包core下面的TopicCommand.scala文件，很容易看出前面都是对命令行参数的校验，然后就是调用createTopic(zkClient, opts)，该函数进入else代码块，对其中的必须的partitions分区数和replicas副本数进行检查，最终调用
AdminUtils.createTopic(zkClient, topic, partitions, replicas, configs)，我们来看下AdminUtils.scala里的具体代码实现：

  def createTopic(zkClient: ZkClient,
                  topic: String,
                  partitions: Int,
                  replicationFactor: Int,
                  topicConfig: Properties = new Properties) {
         val brokerList = ZkUtils.getSortedBrokerList(zkClient)
         val replicaAssignment = AdminUtils.assignReplicasToBrokers(brokerList, partitions, replicationFactor)
         AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(zkClient, topic, replicaAssignment, topicConfig)

  }

1.先从zookeeper的"/brokers/ids"路径获取当前集群的broker id列表并排序
2.AdminUtils.assignReplicasToBrokers是创建topic整个过程中比较重要的算法实现，我们来看看具体代码：

  def assignReplicasToBrokers(brokerList: Seq[Int],
                              nPartitions: Int,
                              replicationFactor: Int,
                              fixedStartIndex: Int = -1,
                              startPartitionId: Int = -1)
    : Map[Int, Seq[Int]] = {
    if (nPartitions <= 0)
        throw new AdminOperationException("number of partitions must be larger than 0")
    if (replicationFactor <= 0)
      throw new AdminOperationException("replication factor must be larger than 0")
    if (replicationFactor > brokerList.size)
      throw new AdminOperationException("replication factor: " + replicationFactor +
        " larger than available brokers: " + brokerList.size)
    val ret = new mutable.HashMap[Int, List[Int]]()
    val startIndex = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerList.size)
    var currentPartitionId = if (startPartitionId >= 0) startPartitionId else 0

    var nextReplicaShift = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerList.size)
    for (i <- 0 until nPartitions) {
      if (currentPartitionId > 0 && (currentPartitionId % brokerList.size == 0))
        nextReplicaShift += 1
      val firstReplicaIndex = (currentPartitionId + startIndex) % brokerList.size
      var replicaList = List(brokerList(firstReplicaIndex))
      for (j <- 0 until replicationFactor - 1)
        replicaList ::= brokerList(replicaIndex(firstReplicaIndex, nextReplicaShift, j, brokerList.size))
      ret.put(currentPartitionId, replicaList.reverse)
      currentPartitionId = currentPartitionId + 1
    }
    ret.toMap
  }

可以看出输入需要的分区数、副本数和集群机器ID列表然后通过计算获取到partition->[replicas in broker] 这样一个Map。这个算法给partition分配broker的实现过程大致是，先从broker列表随机抽取一个分配给partition，然后依次通过移位把下一个broker分配给该partition其他的replica，这样保证了同一个partition的副本可以分配到不同的机器上(以前在一些blog曾经看到别人说kafka集群会根据机器的各种性能去分配partition,现在看完源码其实发现也没那么夸张233)。

    3.createOrUpdateTopicPartitionAssignmentPathInZK顾名思义就是把获得的配置信息写入zookeeper,需要写入两个地方：
        3.1 writeTopicConfig(zkClient, topic, config)，把自定义的Topic-level配置写到"/config/topics/$TOPIC"路径下
       3.2 writeTopicPartitionAssignment(zkClient, topic, partitionReplicaAssignment, update)，把刚才算法计算出来的map数据写入"/brokers/topics/$TOPIC"路径下

到这里其实整个创建topic的api实现过程已经完成了，可能有些同学会问为什么没有见到操作kafka的过程，因为kafka内部是有线程监听zookeeper的节点状态变化，后面的事情就交给kafka内部去处理。
    总结一下定制创建topic api的整个过程，1.从zookeeper获取机器列表 2.算法分配partitions 3.数据写到zookeeper, 由此可以看出这些步骤根本不需要依赖jvm环境，任何语言都能实现这些功能，虽然kafka是用scala实现的，但是不要第一时间就被语言的门槛所阻碍，有空多点琢磨下其实发现scala也不像想象中那么难，而且尝试扩展下kafka的功能也是一件相当好玩的事情。同理，kafka里面的很多api我们都可以用不同的语言去实现。
   下面附上我用node.js实现的kafka创建topic的api代码，这里我根据需求只设计分区数和备份数两个参数，如果需要加上Topic-level的配置可以直接修改传多一个参数即可，
这段代码在kafka后面的新版本同样适用。PS: 我是一直使用scala和java的，从没接触过node.js，我是一边查语法一边写，花了几个小时完成的，写的不好的地方请批评指出，

大家一起学习，一起进步，thx。

代码区------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

var zookeeper = require('node-zookeeper-client');
var Q = require('q');


var BrokerTopicsPath = "/brokers/topics";
var client = zookeeper.createClient('192.168.0.212:2181/kafka');
client.connect();

createTopic('s3', 2, 2);

function createTopic(topic, partitions, replicas) {
    if(-1 !== topic.indexOf("_") || -1 !== topic.indexOf(".")) {
        console.log("WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.")
    }
    if(undefined === partitions || undefined === replicas || 0 >= partitions || 0 >= replicas) {
        console.log("Partitions Or Replicas Must Greater than 0");
        return;
    }
    //Use Kafka AdminUtils
    var brokerList = [];
    Q.fcall(function(){
        var deferred = Q.defer();
        client.getChildren('/brokers/ids', function (error, children, stats) {
            if (error) {
                deferred.reject(new Error(error));
                return;
            }
            deferred.resolve(children);
        });
        return deferred.promise;
    }).then(function (data) {
        brokerList = data.map(function (item) {
            return parseInt(item, 10)
        }).sort();
        console.log("Currently Kafka Cluster Broker: %j", brokerList);
        var replicaAssignment = assignReplicasToBrokers(brokerList, partitions, replicas);
        console.log("ReplicaAssignment Calculate Result: %j", replicaAssignment);
        createOrUpdateTopicPartitionAssignmentPathInZK(topic, replicaAssignment);
    }).catch(function (error) {
        console.log("createTopic Error Occur: " + error)
    });
}

function createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaAssignment) {
    var uniqueMap = {};
    for(var key in partitionReplicaAssignment) {
        var replicaList = partitionReplicaAssignment[key];
        uniqueMap[replicaList.length] = "";
        
        //replica must in difference broker
        var replicaMap = {};
        for(var reKey in replicaList) {
            replicaMap[replicaList[reKey]] = "";
        }
        if(Object.keys(replicaMap).length != replicaList.length) {
            console.log("Duplicate replica assignment found: %j", partitionReplicaAssignment);
            return;
        }
    }
    if(1 != Object.keys(uniqueMap).length) {
        console.log("All partitions should have the same number of replicas.");
        return;
    }
    var topicPath = BrokerTopicsPath + "/" + topic;
    client.exists(topicPath, function (error, stat) {
        if (error) {
            console.log(error.stack);
            return;
        }
        if (stat) {
            console.log('Topic %s already exists.', topic);
        } else {
            writeTopicConfig(topic);
            writeTopicPartitionAssignment(topic, partitionReplicaAssignment);
        }
    });
}

function writeTopicPartitionAssignment(topic, replicaAssignment) {
    var topicPath = BrokerTopicsPath + "/" + topic;
    var jsonPartitionData = {'version': 1, 'partitions': replicaAssignment};

    client.create(topicPath, new Buffer(JSON.stringify(jsonPartitionData), "utf-8"), zookeeper.CreateMode.PERSISTENT,
        function (error, path) {
            if (error) {
                console.log(error.stack);
                return;
            }
            console.log('Success To Create ZNode: %s', path);
        }
    );
}

function writeTopicConfig(topic) {
    var configMap = {"version" : 1,"config" : {}};
    var topicConfigPath = "/config/topics/" + topic;

    client.exists(topicConfigPath, function (error, stat) {
        if (error) {
            console.log(error.stack);
            return;
        }
        if (stat) {
            client.setData(topicConfigPath, new Buffer(JSON.stringify(configMap), "utf-8"), -1, function (error, stat) {
                if (error) {
                    console.log(error.stack);
                    return;
                }
                console.log('%s Update Success', topic);
            });
        } else {
            client.create(topicConfigPath, new Buffer(JSON.stringify(configMap), "utf-8"), zookeeper.CreateMode.PERSISTENT,
                function (error, path) {
                    if (error) {
                        console.log(error.stack);
                        return;
                    }
                    console.log('Success To Create ZNode: %s', path);
                }
            );
        }
    });
}

function assignReplicasToBrokers(brokerList, nPartitions, replicationFactor) {
    var fixedStartIndex = -1;
    var startPartitionId = -1;
    if (replicationFactor > brokerList.length) {
        console.log("replication factor: " + replicationFactor + " larger than available brokers: " + brokerList.length);
        return {};
    }
    var ret = {};
    var startIndex = (fixedStartIndex >= 0) ? fixedStartIndex : getRandomInt(0, brokerList.length);
    var currentPartitionId = (startPartitionId >= 0) ? startPartitionId : 0;
    var nextReplicaShift =  (fixedStartIndex >= 0) ? fixedStartIndex : getRandomInt(0, brokerList.length);
    
    for(var i = 0; i < nPartitions; i++) {
        if (currentPartitionId > 0 && (currentPartitionId % brokerList.length == 0)) {
            nextReplicaShift++
        }
        var firstReplicaIndex = (currentPartitionId + startIndex) % brokerList.length;
        var replicaList = [brokerList[firstReplicaIndex]];
        
        for (var j = 0; j < replicationFactor - 1; j++) {
            replicaList.unshift(brokerList[replicaIndex(firstReplicaIndex, nextReplicaShift, j, brokerList.length)]);
        }
        ret[currentPartitionId] = replicaList.reverse();
        currentPartitionId++;
    }
    return ret;
}

function replicaIndex(firstReplicaIndex, secondReplicaShift, replicaIndex, nBrokers) {
    var shift = 1 + (secondReplicaShift + replicaIndex) % (nBrokers - 1);
    return ((firstReplicaIndex + shift) % nBrokers);
}

function getRandomInt(min, max) {
    min = Math.ceil(min);
    max = Math.floor(max);
    return Math.floor(Math.random() * (max - min)) + min;
}