同一节点启动多个bigchaindb实例

需求

某个节点处于两个不同的bigchaindb集群，或者说一个节点需要启动两个不同的bigchaindb实例

问题

当前bigchaindb版本

BigchainDB (1.0.0rc1)
bigchaindb-driver (0.3.1)

原本的bigchiandb配置文件（/root/.bigchaindb）

{
    "database": {
        "connection_timeout": 5000,
        "max_tries": 3,
        "replicaset": "bigchain-rs",
        "name": "bigchain",
        "host": "localhost",
        "login": null,
        "password": null,
        "ssl": false,
        "port": 27017,
        "backend": "mongodb"
    },
    "log": {
        "level_logfile": "info",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "granular_levels": {},
        "file": "/root/bigchaindb.log",
        "fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "error_file": "/root/bigchaindb-errors.log",
        "fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "level_console": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S"
    },
    "keypair": {
        "private": "BAjL63tqdknLbhXB9vP9F9sgaA699z1CgsMKSKjzFsAg",
        "public": "PKEsj5XCPZjbbsRraixPriPdJ3j9wm1GrMxkcNUAR5n"
    },
    "wsserver": {
        "port": 9985,
        "host": "localhost"
    },
    "server": {
        "threads": null,
        "workers": null,
        "loglevel": "info",
        "bind": "localhost:9984"
    },
    "keyring": [],
    "backlog_reassign_delay": 120
}

利用该配置文件启动一个实例

bigchaindb -c /root/.bigchaindb start

复制一份配置文件，修改其中需要的端口（9984->9982,9985->9983），

cp .bigchaindb .bigchaindb2
sed -i "s/9985/9983/g" .bigchaindb2
sed -i "s/9984/9982/g" .bigchaindb2

再启动一个实例时报错：

root@clean:~# bigchaindb -c .bigchaindb2 start
INFO:bigchaindb.config_utils:Configuration loaded from `.bigchaindb2`
Traceback (most recent call last):
  File "/usr/local/bin/bigchaindb", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/bigchaindb.py", line 348, in main
    utils.start(create_parser(), sys.argv[1:], globals())
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 204, in start
    return func(args)
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 49,in configure
    command(args)
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 75,in start_logging
    setup_logging(user_log_config=config.get('log'))
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 49, in setup_logging
    setup_sub_logger(user_log_config=user_log_config)
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 38, in setup_sub_logger
    server = LogRecordSocketServer()
  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 156, in __init__
    super().__init__((host, port), handler)
  File "/usr/lib/python3.4/socketserver.py", line 430, in __init__
    self.server_bind()
  File "/usr/lib/python3.4/socketserver.py", line 444, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
root@clean:~#

方案

定位到错误位置：/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py

log的setup函数为setup_logging，其中启动了publisher与subscriber。

def setup_logging(*, user_log_config=None):
    setup_pub_logger()
    setup_sub_logger(user_log_config=user_log_config)

log的publisher调用logging.handlers.SocketHandler将远程输出日志到TCP/IP sockets。注意其中使用了写死的端口DEFAULT_SOCKET_LOGGING_PORT

DEFAULT_SOCKET_LOGGING_HOST = 'localhost'
DEFAULT_SOCKET_LOGGING_PORT = DEFAULT_TCP_LOGGING_PORT

def setup_pub_logger():
    dictConfig(PUBLISHER_LOGGING_CONFIG)
    socket_handler = logging.handlers.SocketHandler(
        DEFAULT_SOCKET_LOGGING_HOST, DEFAULT_SOCKET_LOGGING_PORT)
    socket_handler.setLevel(logging.DEBUG)
    logger = logging.getLogger()
    logger.addHandler(socket_handler)

DEFAULT_TCP_LOGGING_PORT在logging模块handlers.py中定义

DEFAULT_TCP_LOGGING_PORT    = 9020

同时，对于日志的subscriber来说，bigchindb启动一个进程来启动subscriber，进程的target为LogRecordSocketServer.serve_forever

def setup_sub_logger(*, user_log_config=None):
    server = LogRecordSocketServer()
    with server:
        server_proc = Process(
            target=server.serve_forever,
            kwargs={'log_config': user_log_config},
        )
        server_proc.start()

该类的__init__里实际上已经指定了订阅的端口，即logging.handlers.DEFAULT_TCP_LOGGING_PORT

class LogRecordSocketServer(ThreadingTCPServer):
    """
    Simple TCP socket-based logging server.

    """
    allow_reuse_address = True

    def __init__(self,
                 host='localhost',
                 port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
                 handler=LogRecordStreamHandler):
        super().__init__((host, port), handler)

    def serve_forever(self, *, poll_interval=0.5, log_config=None):
        sub_logging_config = create_subscriber_logging_config(
            user_log_config=log_config)
        dictConfig(sub_logging_config)
        try:
            super().serve_forever(poll_interval=poll_interval)
        except KeyboardInterrupt:
            pass

至此，我们可以看到，虽然我们bigchaindb的配置文件进行了修改，使得两个实例的bigchaindb端口没有被占用，但实际上两个实例的log的publisher都采用同一个端口

commands/utils.py调用了setup_logging函数

def start_logging_process(command):

    @functools.wraps(command)
    def start_logging(args):
        from bigchaindb import config
        setup_logging(user_log_config=config.get('log'))
        command(args)
    return start_logging

start_logging_process由run_start通过注解的方式进行调用，并同时调用了configure_bigchaindb，继续跟踪，bigchaindb依次调用了bigchaindb.config_utils.autoconfigure->env_config

@configure_bigchaindb
@start_logging_process
def run_start(args):

根据env_config的注释，该函数将config配置文件加载为map，判断是否存在对应的环境变量，否则赋值为配置文件中的值，并最后将配置信息存储在bigchaindb.config中。至此，我们则可以通过在配置文件中加入一个用来配置publisher与subscriber的端口，再在代码中调用bigchaindb.config即可

解决

修改.bigchaindb配置文件，在key值为log下再增加一项

{
    "database": {
       ...
    },
    "log": {
        "port": 9986,
        "level_logfile": "info",

    ...

修改

def setup_logging(*, user_log_config=None):
    logging.info(">>> publisher port is")
    try:
        import bigchaindb
        global DEFAULT_SOCKET_LOGGING_PORT
        DEFAULT_SOCKET_LOGGING_PORT = bigchaindb.config['log']['port']
    except:
        pass
    logging.info(DEFAULT_SOCKET_LOGGING_PORT)
    setup_pub_logger()
    setup_sub_logger(user_log_config=user_log_config)

def setup_sub_logger(*, user_log_config=None):
    server = LogRecordSocketServer(port=DEFAULT_SOCKET_LOGGING_PORT)
    with server:
        server_proc = Process(
            target=server.serve_forever,
            kwargs={'log_config': user_log_config},
        )
        server_proc.start()

再复制一份配置文件：

cp .bigchaindb .bigchaindb2
sed -i "s/9984/9981/g" .bigchaindb2
sed -i "s/9985/9982/g" .bigchaindb2
sed -i "s/9986/9983/g" .bigchaindb2
# 修改数据库名
sed -i "s/\(\"name\": \"\).*/\1bigchain2\"/g" .bigchaindb2

可以启动两个bigchaindb实例了

bigchaindb -c .bigchaindb start
bigchaindb -c .bigchaindb2 start

使用python连接时分别用9984与9981端口即可