Presto将coordinator,worker,discovery,jmx等服务都抽象成了announcer,然后提交给airlift的discovery进行管理,集群所有节点和服务共享一个discovery,很多轻量的信息通过discovery进行同步。理解announcer对理解Presto的节点管理,信息一致性同步等方面有很大的帮助作用。
1. 集群中announcer的查看
presto中可以直接通过discovery的/v1/service接口查看当前左右的注册到discovery url的announcer,如下是个样例:
curl -X GET http://192.168.1.160:9090/v1/service
{
"environment": "production",
"services": [{
"id": "2c42a3b7-9fac-440a-9fcd-11873e0a502c",
"nodeId": "ffffffff-ffff-ffff-ffff-ffffffffffff01",
"type": "presto-coordinator",
"pool": "general",
"location": "/ffffffff-ffff-ffff-ffff-ffffffffffff01",
"properties": {
"http": "http://192.168.1.160:9090",
"http-external": "http://192.168.1.160:9090"
}
}
...
]
}
v1/service的接口的实现是在io.airlift.discovery.discovery-server中实现的,其实这个接口就是将进程中所有注册到discovery的announcer返回而已:
@GET
@Produces(MediaType.APPLICATION_JSON)
public Services getServices()
{
return new Services(node.getEnvironment(), union(dynamicStore.getAll(), staticStore.getAll()));
}
presto中注册的都是dynamicStore。
2. Presto中announcer注册
在presto进程的启动最后的injector.getInstance(Announcer.class).start(),会将所有绑定到ServiceAnnouncement接口的类实例化,然后add到serviceAnnouncements中,serviceAnnouncements保存着所有的service以及service的properties信息。上边提到的/v1/services接口就是从serviceAnnouncements拿到所有的services。
@Inject
public Announcer(DiscoveryAnnouncementClient announcementClient, Set<ServiceAnnouncement> serviceAnnouncements)
{
requireNonNull(announcementClient, "client is null");
requireNonNull(serviceAnnouncements, "serviceAnnouncements is null");
this.announcementClient = announcementClient;
serviceAnnouncements.forEach(this::addServiceAnnouncement);
executor = new ScheduledThreadPoolExecutor(5, daemonThreadsNamed("Announcer-%s"));
executorMBean = new ThreadPoolExecutorMBean((ThreadPoolExecutor) executor);
}
在presto中,绑定了ServiceAnnouncement接口的模块包括:
io.airlift.jmx.JmxModule
io.airlift.jmx.http.rpc.JmxHttpRpcModule
io.prestosql.server.CoordinatorModule
io.prestosql.server.ServerMainModule
3. announcer使用
以Presto中更新catalog为例,介绍在Presto中如何通过announcer的广播动态更新catalog数据源。
private static void updateConnectorIdAnnouncement(Announcer announcer, CatalogName catalogName, InternalNodeManager nodeManager)
{
// 获取到所有注册到discovery的announcement
ServiceAnnouncement announcement = getPrestoAnnouncement(announcer.getServiceAnnouncements());
//修其中一个announcement的信息
Map<String, String> properties = new LinkedHashMap<>(announcement.getProperties());
String property = nullToEmpty(properties.get("connectorIds"));
Set<String> connectorIds = new LinkedHashSet<>(Splitter.on(',').trimResults().omitEmptyStrings().splitToList(property));
connectorIds.add(catalogName.toString());
properties.put("connectorIds", Joiner.on(',').join(connectorIds));
// update announcement
announcer.removeServiceAnnouncement(announcement.getId());
announcer.addServiceAnnouncement(serviceAnnouncement(announcement.getType()).addProperties(properties).build());
// forceAnnounce 可以实现信息广播,通知到所有的coordinator或者worker更新
announcer.forceAnnounce();
nodeManager.refreshNodes();
}
Presto中如果要动态更新catalog信息,所有的coordinator或者worker都需要更新。如果通过一个coordinator或者worker节点收到更新信息后,再逐个通知其他的节点,将是非常麻烦的一件事情。Presto中巧妙利用了announcer进行统一广播,实现了任意一个节点更新,将更新信息推往announcer,因为其他的节点都共享着同样的地址(一样的discovery url),因此所有节点都能实现同时更新,巧妙地利用了数行代码就实现了一个广播机制。
4. 结语
其实在所有的分布式系统中基本上都要实现广播机制进行信息同步,有依赖zookeeper的,有依赖redis的,有依赖数据库的。Presto中实现了一个基于HTTP的极为轻量级别的广播机制,也是一个很巧妙的设计。