trino on yarn调度到同一机器上多实例端口冲突问题处理

一、问题背景

我们在实现trino on yarn的时候，发现当worker的实例数增多的时候，不同的worker就很容易被yarn调度到同一台机器上执行。进而导致启动端口冲突。进程启动失败。这里很难去查找具体的日志，只能够通过修改脚本打印额外日志定位问题。

二、问题分析and修复

当slider提交任务到yarn上调度trino worker on yarn 的时候，触发调度时，会去触发trino-yarn/package/scripts/presto_server.py 这个脚本的start方法来启动trino实例。这个脚本依赖同级目录下的params.py 参数文件。 我们需要做以下的修改：

• params.py
  修改对应这两项

conf_dir = format("{presto_root}/etc2")
data_dir = format("{presto_root}/data")

除此之外，需要在trino-yarn/package/files/trino-server-381 目录下新建data目录，里面啥都不用放。上面这两项做的修改主要是默认配置会导致这两个路径出现冲突，导致启动失败。这里我直接放到trino运行时目录下了（yarn实例调度时创建的目录），这里就不会出现冲突。

• presto_server.py
  直接替换成下面的代码：

# -*- coding: utf-8 -*-
#!/usr/bin/env python

from resource_management import *
from configure import set_configuration
import os
import socket
import datetime
import time
import random

unilogflag =  random.randrange(1,20000,1)

class PrestoServer(Script):
    def __init__(self, component):
        self.component = component

    def install(self, env):
        self.install_packages(env)
        pass

    def configure(self, root_dir):
        set_configuration(self.component, root_dir=root_dir)
    
    def writelog(self, logmsg):
        curtime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        msg = str(unilogflag)+"____"+str(curtime)+' - '+logmsg
        with open('/tmp/presto.log','a') as f:
            f.writelines(msg+'\n')
            f.close()
        print logmsg
    
    def change_port(self,filepath,port):
        self.writelog("ok find a valid port "+str(port)+" begin change !")
        with open(filepath,'a') as f:
            f.writelines("\nhttp-server.http.port="+str(port))
            f.close()
        self.writelog("finished change  port "+str(port))
        cmd = 'cat '+ filepath + ' | grep port'
        self.printResult(os.popen(cmd).readlines())

    def printResult(self,res):
        for line in res:
            if line=='\n':
                continue
            else:
                self.writelog(line)

    def check_port(self):
        import params
        conf = params.conf_dir+'/config.properties' 
        self.writelog('conf dir '+conf)
        flag = True
        port = random.randrange(7000,30000,1)
        while flag:
            self.writelog(' begin detect port '+str(port))
            res = socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex((socket.gethostbyname(socket.gethostname()), port))
            self.writelog(' finish detect port '+str(port)+' res = '+str(res))
            if res == 0:
                logmsg = ' port_confilct '+ str(port) +' and ip is '+str(socket.gethostbyname(socket.gethostname()))
                self.writelog(logmsg)
                port += random.randrange(1,100,1)
                randtime=random.randrange(1,20,2)
                logmsg = ' begin sleep '+randtime +' change port '+str(port)
                self.writelog(logmsg)
                time.sleep(randtime)
            else:
                self.change_port(conf,port)
                params.presto_server_port = port
                logmsg = ' find_valid_port '+str(port)+' params.presto_server_port '+ str(params.presto_server_port)
                self.writelog(logmsg)
                flag = False
                break

    def start(self, env):
        import params

        env.set_params(params)
        start_msg = 'begin start'
        self.writelog(start_msg)
        root_dir = format('{presto_root}/etc/node.properties')
        self.configure(root_dir=root_dir)
        
        os.symlink(os.path.join(format('{presto_root}'), 'etc'),format('{conf_dir}'))
        self.check_port()
        print 'begin execute presto process' 
        print 'presto launche port is '+str(params.presto_server_port)
        process_cmd = format("PATH={presto_root}/jdk-11.0.14/bin:$PATH {presto_root}/bin/launcher run --node-config {conf_dir}/node.properties --jvm-config {conf_dir}/jvm.config --config {conf_dir}/config.properties >> {log_file} 2>&1")

        Execute(process_cmd,
                logoutput=True,
                wait_for_finish=False,
                pid_file=params.pid_file,
                poll_after=3
                )

    def stop(self, env):
        # Slider doesnt yet support stopping the actual app (PrestoServer) process
        # but only stopping the yarn application. Slider is not wired up yet to call this function.
        pass

    def status(self, env):
        import params

        env.set_params(params)
        check_process_status(params.pid_file)


if __name__ == "__main__":
    self.fail_with_error('Component name missing')

上面的脚本大致看看就知道啥意思。主要说下，start方法就是启动trino实例的地方，在真正启动之前，需要修改下trino/etc下的config.properties文件的http-server.http.port启动端口属性。我直接以追加的方式覆盖即可。上面的代码采用了重试+随机端口的方式解决了trino-yarn方式多个实例在同一台机器上启动出现的port冲突问题。