12、网站流量统计

一、网站运行环境配置

在eclipse配置jdk与Tomcat

（一）配置jdk

（二）配置tomcat

（三）分别创建两个web Project

（四）收集的信息

dest=dest_path+"url="+url+"&urlname="+urlname+"&title="+title+"&chset="+charset+"&scr="+screen+"&col="+color+"&lg="+language+"&je="+jvm_enabled+"&ce="+cookie_enabled+"&fv="+flash_ver+"&cnv="+String(Math.random())+"&ref="+ref+"&uagent="+agent+"&stat_uv="+uv_id+"&stat_ss="+stat_ss;

二、离线流处理（未写完）

（一）Flume日志收集的开发

1．发送日志到Flume

在日志服务器中，通过Log4jAppender将日志发往flume客户端
a.在日志服务器应用中导入Log4jAppernder相关开发包
b配置log4i配置文件，实现发送日志给flume

log4j.rootLogger = info,stdout,flume

log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %m%n

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 192.168.232.129
log4j.appender.flume.Port = 44444
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %m%n

c.在日志服务器的LogServlet中，通过log4j来发送日志

logger -into (line)；

2.开发客户端Agent

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444

# shengming sinks
a1.sources.k1.type = avro
a1.sources.k1.hostname = hadoop01
a1.sources.k1.port = 44444

a1.sources.k2.type = avro
a1.sources.k2.hostname = hadoop01
a1.sources.k2.port = 44444

a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balancea1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector = random

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

3、开发中心服务器agent

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 #k2
a1.channels = c1 #c2 

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444

# shengming sinks
a1.sources.k1.type = hdfs
a1.sources.k1.hdfs.path = hdfs://hadoop:9000/flux


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

三、实时流处理（未写完）

（一）修改Flume中心服务器配置

增加kafkaSink，将数据实时导入kafka
hadoop01 hadoop02

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = timestamp

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://hadoop01:9000/reportTime=%Y-%m-%d
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.filePrefix = log_%Y%m%d_%H

a1.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k2.topic=fluxTopic
a1.sinks.k2.brokeList=hadoop01:9092,hadoop02:9092,hadoop03:9092

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

（二）配置Kafka

创建对应主题：
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fluxTopic
启动消费者测试：
kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic fluxTopic

四、遇到错误及解决方法

双击选中内容

修改