How to create a Flume plugin that can listen to any incoming event and alter events’ content on-the-fly ? Using interceptors as described in Flume-interceptors.
But what is a flume Event, and what an Event interceptor might look like ?
Interceptor implementation
First, one needs to define what’s an event. According to flume definition, “A Flume event is defined as a unit of data flow having a byte payload and an optional set of string attributes.[…]”. Its JSON representation could be the following:
{"headers":{"k1":"v1"},"body":"this is my body"}
We wish to alter incoming flow and enrich with custom headers (custom attributes) as follows:
{"headers":{"k1":"v1","k2","v2"},"body":"this is my body"}
Create project and dependencies
Let’s create a simple maven project with the following dependency. I am currently using version 1.4.0-cdh4.6.0 (cloudera release) of Flume, but the implementation should be the same regardless the flume version.
1
2
3
4
5
|
<
dependency
>
<
groupId
>org.apache.flume</
groupId
>
<
artifactId
>flume-ng-core</
artifactId
>
<
version
>${flume.core.version}</
version
>
</
dependency
>
|
Build interception class
I created a CustomHostInterceptor as described below. The goal of this interceptor is to enrich incoming data with hostname value. This, obviously, is already packaged in flume libraries (org.apache.flume.interceptor.HostInterceptor$Builder), but this will give you a good example of how the interception works.
Here is what the Interceptor interface looks like. This will be implemented by ourCustomHostInterceptor.class.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
public
interface
Interceptor {
void
initialize();
Event intercept(Event event);
List<Event> intercept(List<Event> list);
void
close();
static
interface
Builder
extends
Configurable {
Interceptor build();
}
}
|
static Builder will read properties from Flume context and initialize Interceptor class accordingly. The interceptor logic itself happens in intercept(Event event) method. Since flume is able to handle batch of events, we therefore have to handle batch of interceptions (hence the List intercept(List list) method).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
|
public
class
CustomHostInterceptor
implements
Interceptor {
private
String hostValue;
private
String hostHeader;
public
CustomHostInterceptor(String hostHeader){
this
.hostHeader = hostHeader;
}
@Override
public
void
initialize() {
// At interceptor start up
try
{
hostValue =
InetAddress.getLocalHost().getHostName();
}
catch
(UnknownHostException e) {
throw
new
FlumeException(
"Cannot get Hostname"
, e);
}
}
@Override
public
Event intercept(Event event) {
// This is the event's body
String body =
new
String(event.getBody());
// These are the event's headers
Map<String, String> headers = event.getHeaders();
// Enrich header with hostname
headers.put(hostHeader, hostValue);
// Let the enriched event go
return
event;
}
@Override
public
List<Event> intercept(List<Event> events) {
List<Event> interceptedEvents =
new
ArrayList<Event>(events.size());
for
(Event event : events) {
// Intercept any event
Event interceptedEvent = intercept(event);
interceptedEvents.add(interceptedEvent);
}
return
interceptedEvents;
}
@Override
public
void
close() {
// At interceptor shutdown
}
public
static
class
Builder
implements
Interceptor.Builder {
private
String hostHeader;
@Override
public
void
configure(Context context) {
// Retrieve property from flume conf
hostHeader = context.getString(
"hostHeader"
);
}
@Override
public
Interceptor build() {
return
new
CustomHostInterceptor(hostHeader);
}
}
}
|
No need to spend time explaining what’s happening there, pretty self-explanatory. The Hostname will be written on event’s header under a custom key (supplied through the flume configuration using key “hostHeader”).
Package / Install Interceptor
In 3 words : mvn clean package
This above cmd will package your interceptor into a single JAR file that can be read at flume-agent start up. Your jar file must be uploaded on each server running flume-agent. Default path for any flume plugin is /usr/lib/flume-ng/lib/. Path might be slightly different when using Cloudera-manager. The interceptor can be used, but needs to be activated first from the flume configuration (/etc/flume-ng/conf/flume.conf).
# Describe sources ########################### a1.sources = r1 # .../... # Attached the interceptor to the source a1.sources.r1.interceptors = i1 a1.sources.r1.interceptors.i1.type = com.aamend.hadoop.flume.interceptor.CustomHostInterceptor$Builder a1.sources.r1.interceptors.i1.hostHeader = hostname # Describe sinks ########################### # .../... # Describe channels ########################### # .../...
Once properly configured, restart the flume-ng daemon and monitor the flume log file (/var/log/flume-ng/flume.log).
Any incoming event should now have been enriched with hostname. Enabling Logging as per the source code available on my GitHub account, you should see something like:
2014-06-11T13:04:48+0100 INFO .flume.interceptor.CustomHostInterceptor: 50 - Before interception : {"headers":{"k1":"v1"},"body":"this is my body"} 2014-06-11T13:04:48+0100 INFO .flume.interceptor.CustomHostInterceptor: 65 - After interception : {"headers":{"hostname":"localhost.localdomain","k1":"v1"},"body":"this is my body"}
Congratulations, your interceptor works. You can then alter any incoming events, enrich headers, or even redirect to different channels based on header’s values (seemultiplexing). The latter will be a matter of discussion in a next post.
Reference: https://hadoopi.wordpress.com/2014/06/11/flume-getting-started-with-interceptors/