要实现StormFirehouse 首先需要在Druid端注册一个StormFirehoseFactory,其实现就是实现FirehoseFactory接口,我们需要注意StormFirehosFactory注册对象是json类型的,必须要在类名上写上注解@JsonTypeName(“storm”)
从代码看StormFirehoseFactor主要就是就是生成并且返回Firehose对象.
@JsonTypeName("storm")
public class StormFirehoseFactory implements FirehoseFactory {
private static final StormFirehose FIREHOSE = new StormFirehose();
@JsonCreator
public StormFirehoseFactory() {
}
public static StormFirehose getFirehose() {
return FIREHOSE;
}
@Override
public Firehose connect(InputRowParser inputRowParser) throws IOException, ParseException {
return FIREHOSE;
}
}
StormFirehose继承实现Firehose接口,主要实现了hasMore ,nextRow,Commit方法
在类中定义了全局静态的数据队列,其中BLOCK_QUEUE使用来存储DruidState中需要写入到Druid中的数据,也就是之前Storm需要持久化的数据持久化到StormFirehose的BlockQueue中,Druid掉用hasMore轮训对列是否为空,如果对为,方法会调用START.wait()一直阻塞在这里,如果BLOCK_QUEUE对垒的数据部位空,调用hasMore就会返回True,这是Druid会接着调用nextRow,如果对垒中的数据消费完成,那么当前批次提交的数据,已经处理完成,这个时候,需要把当前批次的ID保存到LIMBO_TRANSACTIOS对列中,同事在判断BLOCK_QUEUE对列中的数据为空的时候,可以释放sendMessage的阻塞状态,使得数据可以重新写入到对列中.然后Druid调用commit方法后,该方法需要返回一个Rannable对象,Durid会启动改线程,完成将LIMBO_TRANSACTIONS对列中的partitionId修改为COMPLETE状态.对于StormFirehose中自己写的sendMessage方法,其功能主要是把DruidState中的需要写入Druid的数据写入到BLOCK_QUEUE对列中,并且释放hasMore方法阻塞,并且阻塞改方法,一面外部数据再次提交.
“
public class StormFirehose implements Firehose {
private static final Logger LOG = LoggerFactory.getLogger(StormFirehose.class);
private static final Object START = new Object();
private static final Object FINISHED = new Object();
//数据队列
private static BlockingQueue BLOCKING_QUEUE;
public static final DruidPartitionStatus STATUS = new DruidPartitionStatus();
private static String TRANSACTION_ID = null;
private static BlockingQueue LIMBO_TRANSACTIONS = new ArrayBlockingQueue(99999);
@Override
public boolean hasMore() {
if (BLOCKING_QUEUE != null && !BLOCKING_QUEUE.isEmpty()) {
return true;
}
try {
synchronized (START) {
START.wait();
}
} catch (InterruptedException e) {
LOG.error("hasMore() blocking was interrupted.", e);
}
return true;
}
@Override
public InputRow nextRow() {
final Map<String, Object> theMap = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
try {
FixMessageDto message;
message = BLOCKING_QUEUE.poll();
if (message != null) {
// LOG.info("[" + message.symbol + "] @ [" + message.price + "] for [" + message.uid + "]");
theMap.put("symbol", message.symbol);
theMap.put("price", message.price);
}
if (BLOCKING_QUEUE.isEmpty()) {
STATUS.putInLimbo(TRANSACTION_ID);
LIMBO_TRANSACTIONS.add(TRANSACTION_ID);
LOG.info("Batch is fully consumed by Druid. Unlocking [FINISH]");
synchronized (FINISHED) {
FINISHED.notify();
}
}
} catch (Exception e) {
LOG.error("Error occurred in nextRow.", e);
}
final LinkedList<String> dimensions = new LinkedList<String>();
dimensions.add("symbol");
dimensions.add("price");
return new MapBasedInputRow(System.currentTimeMillis(), dimensions, theMap);
}
@Override
public Runnable commit() {
List<String> limboTransactions = new ArrayList<String>();
LIMBO_TRANSACTIONS.drainTo(limboTransactions);
return new StormCommitRunnable(limboTransactions);
}
public synchronized void sendMessages(String partitionId, List<FixMessageDto> messages) {
BLOCKING_QUEUE = new ArrayBlockingQueue<FixMessageDto>(messages.size(), false, messages);
TRANSACTION_ID = partitionId;
LOG.info("Beginning commit to Druid. [" + messages.size() + "] messages, unlocking [START]");
//对象锁
synchronized (START) {
START.notify();
}
try {
synchronized (FINISHED) {
FINISHED.wait();
}
} catch (InterruptedException e) {
LOG.error("Commit to Druid interrupted.");
}
LOG.info("Returning control to Storm.");
}
@Override
public void close() throws IOException {
// do nothing
}
}
StormCommitRunnable 是一个线程,在StormFirehose的commit方法调用时候,调用该线程主要是调用了DruidPartitionStatus类的complete方法,将limbo的批次的数据修改为已处理完成.
public class StormCommitRunnable implements Runnable {
private List partitionIds = null;
public StormCommitRunnable(List<String> partitionIds) {
this.partitionIds = partitionIds;
}
@Override
public void run() {
try {
StormFirehose.STATUS.complete(partitionIds);
} catch (Exception e) {
Log.error("Could not complete transactions.", e);
}
}
}
DruidPartitionStatus类中的各个方法主要是更改每个批次的数据在zk中标记的状态.
public class DruidPartitionStatus {
private static final Logger LOG = LoggerFactory.getLogger(DruidPartitionStatus.class);
final String COMPLETED_PATH = “completed”;
final String LIMBO_PATH = “limbo”;
final String CURRENT_PATH = “current”;
private CuratorFramework curatorFramework;
public DruidPartitionStatus() {
try {
curatorFramework = CuratorFrameworkFactory.builder().namespace("stormdruid")
.connectString("localhost:2181").retryPolicy(new RetryNTimes(1, 1000)).connectionTimeoutMs(5000)
.build();
curatorFramework.start();
if (curatorFramework.checkExists().forPath(COMPLETED_PATH) == null) {
curatorFramework.create().forPath(COMPLETED_PATH);
}
if (curatorFramework.checkExists().forPath(CURRENT_PATH) == null) {
curatorFramework.create().forPath(CURRENT_PATH);
}
if (curatorFramework.checkExists().forPath(LIMBO_PATH) == null) {
curatorFramework.create().forPath(LIMBO_PATH);
}
} catch (Exception e) {
LOG.error("Could not establish conneciton to Zookeeper", e);
}
}
public boolean isCompleted(String partitionId) throws Exception {
return (curatorFramework.checkExists().forPath(COMPLETED_PATH + "/" + partitionId) != null);
}
public boolean isInLimbo(String partitionId) throws Exception {
return (curatorFramework.checkExists().forPath(LIMBO_PATH + "/" + partitionId) != null);
}
public boolean isInProgress(String partitionId) throws Exception {
return (curatorFramework.checkExists().forPath(CURRENT_PATH + "/" + partitionId) != null);
}
public void putInProgress(String partitionId) throws Exception {
curatorFramework.create().forPath(CURRENT_PATH + "/" + partitionId);
}
public void putInLimbo(String partitionId) throws Exception {
curatorFramework.inTransaction().
delete().forPath(CURRENT_PATH + "/" + partitionId)
.and().create().forPath(LIMBO_PATH + "/" + partitionId).and().commit();
}
public void complete(List<String> partitionIds) throws Exception {
Iterator<String> iterator = partitionIds.iterator();
CuratorTransaction transaction = curatorFramework.inTransaction();
while (iterator.hasNext()) {
String partitionId = iterator.next();
transaction = transaction.delete().forPath(LIMBO_PATH + "/" + partitionId)
.and().create().forPath(COMPLETED_PATH + "/" + partitionId).and();
}
CuratorTransactionFinal tx = (CuratorTransactionFinal) transaction;
tx.commit();
}
}
至此StormFirehose的实现结束,那么StormFirehose是如何完成数据摄取的呢,在之前的Trident Topology中我们是用了TridentTopology的persistent方法将数据通过DruidBeam的方式持久化到druid中,那么这里StormFirehose的与DruidBeam的功能类似,唯一不同的是druid的durid数据摄取方式的不同,在DruidBeam中是使用push的方式来摄取数据的,在StormFirehos中是通过pull的方式来实现数据摄取的,在Storm基本上是类似的,下面我么看看是入和实现的:
首先需要一个TridentTopology来调用persistent方法,第一个参数是StateFactory实例,用于创建DrudiStat实例,并且调用DruidState中的方法将数据持久化到其他的系统中.第二个参数是需要接受的字段名称,第三个字段是StateUpdate将数据,解析接收到的tuple数据,并且在调用StateFactory实例中的额方法,将数据存储待其他系统中.
具体代码如下:
FinancialAnalyticsTopology核心方法是
inputStream.each(new Fields("message"), new MessageTypeFilter())
.partitionPersist(new DruidStateFactory(), new Fields("message"), new DruidStateUpdater());
return topology.build();
partitionPersist方法,将数据持久化到其他的系统中.
public class FinancialAnalyticsTopology {
private static final Logger LOG = LoggerFactory.getLogger(FinancialAnalyticsTopology.class);
public static StormTopology buildTopology() {
LOG.info("Building topology.");
TridentTopology topology = new TridentTopology();
FixEventSpout spout = new FixEventSpout();
Stream inputStream = topology.newStream("message", spout);
inputStream.each(new Fields("message"), new MessageTypeFilter())
.partitionPersist(new DruidStateFactory(), new Fields("message"), new DruidStateUpdater());
return topology.build();
}
public static void main(String[] args) throws Exception {
/*LogLevelAdjuster.register();*/
final Config conf = new Config();
final LocalCluster cluster = new LocalCluster();
LOG.info("Submitting topology.");
cluster.submitTopology("financial", conf, buildTopology());
LOG.info("Topology submitted.");
Thread.sleep(600000);
}
}
DruidStateFactory必须继承StateFactory并实现mekeState放创建实现State接口的DrudState方法
@SuppressWarnings(“rawtypes”)
public class DruidStateFactory implements StateFactory {
private static final long serialVersionUID = 1L;
private static final Logger LOG = LoggerFactory.getLogger(DruidStateFactory.class);
@Override
public State makeState(Map conf, IMetricsContext metrics, int partitionIndex, int numPartitions) {
return new DruidState(partitionIndex);
}
}
DruidState在该方法中Strom主要是通过事物的方式将数据提交给其他的外部存储系统处理
public class DruidState implements State {
private static final Logger LOG = LoggerFactory.getLogger(DruidState.class);
private Vector messages = new Vector();
private int partitionIndex;
public DruidState(int partitionIndex){
this.partitionIndex = partitionIndex;
}
@Override
public void beginCommit(Long batchId) {
}
@Override
public void commit(Long batchId) {
String partitionId = batchId.toString() + "-" + partitionIndex;
LOG.info("Committing partition [" + partitionIndex + "] of batch [" + batchId + "]");
try {
if (StormFirehose.STATUS.isCompleted(partitionId)) {
LOG.warn("Encountered completed partition [" + partitionIndex + "] of batch [" + batchId + "]");
return;
} else if (StormFirehose.STATUS.isInLimbo(partitionId)) {
LOG.warn("Encountered limbo partition [" + partitionIndex + "] of batch [" + batchId + "] : NOTIFY THE AUTHORITIES!");
return;
} else if (StormFirehose.STATUS.isInProgress(partitionId)) {
LOG.warn("Encountered in-progress partition [\" + partitionIndex + \"] of batch [" + batchId + "] : NOTIFY THE AUTHORITIES!");
return;
}
StormFirehose.STATUS.putInProgress(partitionId);
StormFirehoseFactory.getFirehose().sendMessages(partitionId, messages);
} catch (Exception e) {
LOG.error("Could not start firehose for [" + partitionIndex + "] of batch [" + batchId + "]", e);
}
}
public void aggregateMessage(FixMessageDto message) {
// LOG.info("Aggregating [" + message + "]");
messages.add(message);
}
}
StateUpdater 对于每次接受到的tuple数据,进行处理,并且调用DruidState的方法保存到其他的外部存储系统中
public class DruidStateUpdater implements StateUpdater {
private static final long serialVersionUID = 1L;
// private static final Logger LOG = LoggerFactory.getLogger(DruidStateUpdater.class);
@SuppressWarnings("rawtypes")
@Override
public void prepare(Map conf, TridentOperationContext context) {
}
@Override
public void cleanup() {
}
@Override
public void updateState(DruidState state, List<TridentTuple> tuples, TridentCollector collector) {
//LOG.info("Updating [" + state + "]");
for (TridentTuple tuple : tuples) {
FixMessageDto message = (FixMessageDto) tuple.getValue(0);
state.aggregateMessage(message);
}
}
}
“`