storm-[7]-Trident State学习

Trident State

Trident 提供以下机制实现“exactly-once processing”：

1.Tuples处理基于小的batches

2.每一个batch分配一个唯一的"transaction id" (txid)，重试时该id不变

3.State updates在batches间是有序的（the state updates for batch 3 won't be applied until the state updates for batch 2 have succeeded.）

同时，fault-tolerance动作依赖于输入spout提供的semantics，有三种spouts："non-transactional", "transactional", and "opaque transactional"

事务型，非事务型，非透明事务型

Transactional spouts

每一个batch分配一个唯一的"transaction id" (txid)，重试时该id不变，这一特性依赖于spouts的特性。 transactional spout:

1. Batches for a given txid are always the same. Replays of batches for a txid will exact same set of tuples as the first time that batch was emitted for that txid.
2. There's no overlap between batches of tuples (tuples are in one batch or another, never multiple).
3. Every tuple is in a batch (no tuples are skipped)
   
   

这是最单易懂的实现方式，为什么还会用其他方式的spout呢？原因之一是以上规则决定了一些情况下存在非容错性。例如：TransactionalTridentKafkaSpout 为Kafka主题分区中的所有tuples生成txid，一旦其中一个bacth发射后，若任何时刻该batch需重发，根据以上规则那么所有该集合中的tuples都要重新emitted一次。假设一个batch发送后还未被处理，此时一个Kafka的节点挂掉了，该情形下就不能像以前一样replay一个同样的batch，处理过程将halt。

 一个实现"transactional state"的例子，单词计数：

试想，如果只存（key，value）也即（word，count）将无法知道我们是否处理过待处理的tuple，可以如下方式加以解决，

1.count和transaction id一起在数据库中已原子vaule的方式存储

2.当要更新count时，检测数据库中的transaction id 和当前batch的transaction id 是否一致，若一致则忽略本次更新动作（因为已处理过），若不同则对count做increment更新

具体实例如下，假设我们要处理txid 3的batch，其tuples如下：

["man"]
["man"]
["dog"]

假设当前数据库中已存在的key/value对如下：

man => [count=3, txid=1]
dog => [count=4, txid=3]
apple => [count=10, txid=2]

检测到"man"的txid 1，txid 3的["man"]没有处理过则，更新如下：

man => [count=5, txid=3]
dog => [count=4, txid=3]
apple => [count=10, txid=2]

Opaque transactional spouts

opaque transactional spout不保证batch中的tuples拥有不变的txid，opaque transactional spout 特性如下：

保证每一个tuple在一个batch中被处理，也即若一个tuple在一个batch中处理失败，则允许保证在后来的batch中被处理即可。

在opaque transactional spouts中不同于transactional spouts，若 数据库中的transaction id 和当前待处理batch的transaction id相同，则说明batch可能在状态更新间发生过变更。为此，数据库中要存储更多的状态信息做标识，不止存储a value, transaction id，现在存储a value, transaction id, and the previous value。

假设 batch is "2" 的计数结果如下:

{ value = 4, prevValue = 1, txid = 2 } 

假设当前的txid为3，此时将prevValue设为value，并用当前计算值增加到value中，将txid设为3：

{ value = 6, prevValue = 4, txid = 3 } 

相反若假设当前的txid仍为2，和库中一样，我们将得知库value发生过变化了（需要更新重置）：

{ value = 3, prevValue = 1, txid = 2 } 

该中做法有效基于Trident的batches的强有序性（the state updates for batch 3 won't be applied until the state updates for batch 2 have succeeded），以及非重叠性

Non-transactional spouts

非事物型不能保证batch的内容，因此只能提供至多一次的处理（tuple处理失败不重试），或者至少一次的处理（tuples可能在多个batch中处理成功）

Summary of spout and state types

This diagram shows which combinations of spouts / states enable exactly-once messaging semantics:

1. Opaque transactional states have the strongest fault-tolerance, but this comes at the cost of needing to store the txid and two values in the database.
2. Transactional states require less state in the database, but only work with transactional spouts.
3. Finally, non-transactional states require the least state in the database but cannot achieve exactly-once semantics.
4. The state and spout types you choose are a tradeoff between fault-tolerance and storage costs, and ultimately your application requirements will determine which combination is right for you.

State APIs

Trident在State内部实现 fault-tolerance，使用者不必实现诸如：处理比较txids，存储 multiple values等事项。可如下编码：

TridentTopology topology = new TridentTopology();        
TridentState wordCounts =
      topology.newStream("spout1", spout)
        .each(new Fields("sentence"), new Split(), new Fields("word"))
        .groupBy(new Fields("word"))
        .persistentAggregate(MemcachedState.opaque(serverLocations), new Count(), new Fields("count"))                
        .parallelismHint(6);

MemcachedState.opaque 的调用实现所有opaque transactional state 逻辑，此外updates自动的已批处理减少切换互链的形式作用于database。

base State 接口有如下两个方法：

public interface State {
    void beginCommit(Long txid); // can be null for things like partitionPersist occurring off a DRPC stream
    void commit(Long txid);
}

告知使用者state update开始与结束，并且给出每种case下的txid。

假设你有一个home-grown database存储着用户的location信息，你想通过Trident访问它。你的State实现中要实现getting和setting方法来得到用户的信息：

public class LocationDB implements State {
    public void beginCommit(Long txid) {    
    }

    public void commit(Long txid) {    
    }

    public void setLocation(long userId, String location) {
      // code to access database and set location
    }

    public String getLocation(long userId) {
      // code to get location from database
    }
}

然后你要为Trident实现一个StateFactory返回你的状态对象:

public class LocationDBFactory implements StateFactory {
   public State makeState(Map conf, int partitionIndex, int numPartitions) {
      return new LocationDB();
   } 
}

Trident提供一个QueryFunction接口实现Trident写操作，StateUpdater实现更新操作。例如，我们写一个操作 "QueryLocation" 实现查询LocationDB得到用户locations：

TridentTopology topology = new TridentTopology();
TridentState locations = topology.newStaticState(new LocationDBFactory());
topology.newStream("myspout", spout)
        .stateQuery(locations, new Fields("userid"), new QueryLocation(), new Fields("location"))

QueryLocation 的大致实现如下:

public class QueryLocation extends BaseQueryFunction<LocationDB, String> {
    public List<String> batchRetrieve(LocationDB state, List<TridentTuple> inputs) {
        List<String> ret = new ArrayList();
        for(TridentTuple input: inputs) {
            ret.add(state.getLocation(input.getLong(0)));
        }
        return ret;
    }

    public void execute(TridentTuple tuple, String location, TridentCollector collector) {
        collector.emit(new Values(location));
    }    
}

QueryFunction通过2步实现，首先Trident将读取来的batch收集在一起，传递给batchRetrieve，batchRetrieve将接收多个用户的ids。batchRetrieve将返回一个结果list。list中的顺序和输入tuple中一致。

You can see that this code doesn't take advantage of the batching that Trident does, since it just queries the LocationDB one at a time. So a better way to write the LocationDB would be like this:

public class LocationDB implements State {
    public void beginCommit(Long txid) {    
    }

    public void commit(Long txid) {    
    }

    public void setLocationsBulk(List<Long> userIds, List<String> locations) {
      // set locations in bulk
    }

    public List<String> bulkGetLocations(List<Long> userIds) {
      // get locations in bulk
    }
}

Then, you can write the QueryLocation function like this:

public class QueryLocation extends BaseQueryFunction<LocationDB, String> {
    public List<String> batchRetrieve(LocationDB state, List<TridentTuple> inputs) {
        List<Long> userIds = new ArrayList<Long>();
        for(TridentTuple input: inputs) {
            userIds.add(input.getLong(0));
        }
        return state.bulkGetLocations(userIds);
    }

    public void execute(TridentTuple tuple, String location, TridentCollector collector) {
        collector.emit(new Values(location));
    }    
}

This code will be much more efficient by reducing roundtrips to the database.

 

To update state, you make use of the StateUpdater interface. Here's a StateUpdater that updates a LocationDB with new location information:

public class LocationUpdater extends BaseStateUpdater<LocationDB> {
    public void updateState(LocationDB state, List<TridentTuple> tuples, TridentCollector collector) {
        List<Long> ids = new ArrayList<Long>();
        List<String> locations = new ArrayList<String>();
        for(TridentTuple t: tuples) {
            ids.add(t.getLong(0));
            locations.add(t.getString(1));
        }
        state.setLocationsBulk(ids, locations);
    }
}

Here's how you would use this operation in a Trident topology:

TridentTopology topology = new TridentTopology();
TridentState locations = 
    topology.newStream("locations", locationsSpout)
        .partitionPersist(new LocationDBFactory(), new Fields("userid", "location"), new LocationUpdater())

The partitionPersist operation updates a source of state. The StateUpdater receives the State and a batch of tuples with updates to that State. This code just grabs the userids and locations from the input tuples and does a bulk set into the State.

partitionPersist returns a TridentState object representing the location db being updated by the Trident topology. You could then use this state in stateQuery operations elsewhere in the topology.

You can also see that StateUpdaters are given a TridentCollector. Tuples emitted to this collector go to the "new values stream". In this case, there's nothing interesting to emit to that stream, but if you were doing something like updating counts in a database, you could emit the updated counts to that stream. You can then get access to the new values stream for further processing via the TridentState#newValuesStream method.

persistentAggregate

Trident has another method for updating States called persistentAggregate. You've seen this used in the streaming word count example, shown again below:

TridentTopology topology = new TridentTopology();        
TridentState wordCounts =
      topology.newStream("spout1", spout)
        .each(new Fields("sentence"), new Split(), new Fields("word"))
        .groupBy(new Fields("word"))
        .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count"))

persistentAggregate is an additional abstraction built on top of partitionPersist that knows how to take a Trident aggregator and use it to apply updates to the source of state. In this case, since this is a grouped stream, Trident expects the state you provide to implement the "MapState" interface. The grouping fields will be the keys in the state, and the aggregation result will be the values in the state. The "MapState" interface looks like this:

public interface MapState<T> extends State {
    List<T> multiGet(List<List<Object>> keys);
    List<T> multiUpdate(List<List<Object>> keys, List<ValueUpdater> updaters);
    void multiPut(List<List<Object>> keys, List<T> vals);
}

When you do aggregations on non-grouped streams (a global aggregation), Trident expects your State object to implement the "Snapshottable" interface:

public interface Snapshottable<T> extends State {
    T get();
    T update(ValueUpdater updater);
    void set(T o);
}

MemoryMapState and MemcachedState each implement both of these interfaces.

Implementing Map States

Trident makes it easy to implement MapState's, doing almost all the work for you. The OpaqueMap, TransactionalMap, and NonTransactionalMap classes implement all the logic for doing the respective fault-tolerance logic. You simply provide these classes with an IBackingMap implementation that knows how to do multiGets and multiPuts of the respective key/values. IBackingMap looks like this:

public interface IBackingMap<T> {
    List<T> multiGet(List<List<Object>> keys); 
    void multiPut(List<List<Object>> keys, List<T> vals); 
}

OpaqueMap's will call multiPut with OpaqueValue's for the vals, TransactionalMap's will give TransactionalValue's for the vals, and NonTransactionalMaps will just pass the objects from the topology through.

Trident also provides the CachedMap class to do automatic LRU caching of map key/vals.

Finally, Trident provides the SnapshottableMap class that turns a MapState into a Snapshottable object, by storing global aggregations into a fixed key.

Take a look at the implementation of MemcachedState to see how all these utilities can be put together to make a high performance MapState implementation. MemcachedState allows you to choose between opaque transactional, transactional, and non-transactional semantics.