2017年06月_GE12

12月 10月 07月 06月 05月 04月 03月 02月 01月

原创 Spark SQL Hive Tables

Spark SQL also supports reading and writing data stored in Apache Hive. However, since Hive has a large number of dependencies, these dependencies are not included in the default Spark distribution.

2017-06-28 16:43:00 1021

原创 10.Flume Channels

1.Memory Channela1.channels = c1a1.channels.c1.type = memorya1.channels.c1.capacity = 10000a1.channels.c1.transactionCapacity = 10000a1.channels.c1.byteCapacityBufferPercentage = 20a1.channel

2017-06-13 16:04:51 920

原创 9.Flume Sinks

1.Hdfs Sinka1.channels = c1a1.sinks = k1a1.sinks.k1.type = hdfsa1.sinks.k1.channel = c1a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%Sa1.sinks.k1.hdfs.filePrefix = events-a1.sinks.k1.

2017-06-13 15:58:09 1150

原创 Removing Hive Metastore Password from Hive Configuration

1.Set up the CredentialProvider to store the Hive Metastore password, using the key javax.jdo.option.ConnectionPassword (the same key as used in the Hive configuration). $ hadoop credential create j

2017-06-09 13:38:17 920

原创 nginx proxy_next_upstream

2017-06-07 16:45:06 10070

原创 8.Flume Log4J Appender

Log4J AppenderAppends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). Required properties

2017-06-06 17:02:47 1049

原创 7.Flume Interceptors

Flume InterceptorsFlume has the capability to modify/drop events in-flight. This is done with the help of interceptors. Interceptors are classes that implement org.apache.flume.interceptor.Interce

2017-06-06 17:01:12 1010

原创 6.Flume Channel Selectors

Flume Channel SelectorsIf the type is not specified, then defaults to “replicating”.Replicating Channel Selector (default)Required properties are in bold.Property NameDefau

2017-06-06 16:59:50 848

原创 5.Flume 多个channel sinks

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyrigh

2017-06-06 15:13:25 662

原创 4.Flume client 开发

1.下载 Flume client sdkhttp://central.maven.org/maven2/org/apache/flume/flume-ng-sdk/1.7.0/flume-ng-sdk-1.7.0.jar2、RPC client interfaceFlume的RpcClient实现了Flume的RPC机制。用户的应用程序可以很简单的调用Flume Clie

2017-06-06 10:50:20 2093

原创 3. Flume 启动命令

./flume-ng agent --conf ../conf -f ../conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

2017-06-05 15:23:35 5090

原创 2. Flume Source 详解

1. Avro Source属性说明a1.sources = r1a1.channels = c1a1.sources.r1.type = avroa1.sources.r1.channels = c1a1.sources.r1.bind = 0.0.0.0a1.sources.r1.port = 41412. Spooling Directory Source

2017-06-05 13:39:26 2663

原创 1.Apache Flume 简介

一.简介Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。image.png二.主要功能1.日志收集Flume最早是Cloudera提供的日志收集系统，目前是Apac

2017-06-05 10:01:00 1726

原创 tcmalloc安装

一. 安装　　tcmalloc在gperftools之中，故想要使用tcmalloc，就得先安装gperftools。在linux下，其安装步骤如下：　　1. tar xzvf gperftools-2.1.tar.gz　　2. cd gperftools-2.1　　3. ./configure --enable-frame-pointers　　4. mak

2017-06-02 16:20:14 1668

原创 JBoss AS JBoss EAP 和wildfly

JBoss是一个基于JavaEE的开源的应用服务器，遵循LGPL许可，意味着可以在任何商业应用中免费使用。2006年，JBoss被Redhat公司收购。 JBoss AS目前作为Redhat公司的商业产品JBoss Enterprise Application Platform的上游基础，为了使这两个产品有差异化，避免用户混淆，因此该公司在去年10月份就寻求为JBoss AS找一个新名

2017-06-02 13:16:02 3157 3

原创 TTL Indexes

一、 Time To Live(TTL) 集合MongoDB 2.2 引入一个新特性--TTL 集合，TTL集合支持失效时间设置，当超过指定时间后，集合自动清除超时文档，者用来保存一个诸如session会话信息的时候非常有用。如果想使用TTL集合，用用到 expireAfterSeconds 选项 db.ttl.ensureIndex({ "date": new Date() }, {

2017-06-01 13:21:47 1827