一、说明
将此项目打成一个jar包telecomeAnalysis-1.0.0.jar,提交到yarn进行测试;
1、将telecomeAnalysis-1.0.0.jar放到master的/home/test/sparkpro/目录下
2、phoenix先查看表数据为空
3、查看yarn正在运行的任务列表为空
[root@master ~]# yarn application -appStates running -list
20/06/24 15:08:45 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
Total number of applications (application-types: [] and states: [RUNNING]):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
[root@master ~]#
二、将spark程序提交到yarn
1、后台提交kafka2sparkStreaming2Hbase程序
[root@master bin]# nohup ./spark-submit --master yarn --class com.cn.sparkStreaming.kafka2sparkStreaming2Hbase /home/test/sparkpro/telecomeAnalysis-1.0.0.jar &
[1] 4938
2、tail -f nohup.out观察后台日志
3、查看yarn任务列表
[root@master ~]# yarn application -appStates running -list
20/06/24 17:20:30 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
Total number of applications (application-types: [] and states: [RUNNING]):1
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
application_1592989983675_0001 kafka2sparkStreaming2Hbase SPARK root default RUNNING UNDEFINED 10% http://192.168.230.21:4040
三、IDEA运行模拟数据
1、idea运行造数据程序
2、查看phoenix表数据
3、为什么不在spark运行程序的linux上打包运行此程序
由于虚拟机内存有限问题,因为spark程序就已经很吃内存了,如果再运行此程序在同一个linux上,会出现组件由于内存有限的错误信息,导致模拟无法进行;
四、在另一台服务器上运行造数据程序
1、先清除表数据
0: jdbc:phoenix:master,slaves1,slaves2:2181> delete from "location_sure";
5 rows affected (0.012 seconds)
0: jdbc:phoenix:master,slaves1,slaves2:2181> select * from "location_sure";
+--------------+-------------+-------------+
| rowkey_name | count_time | walk_place |
+--------------+-------------+-------------+
+--------------+-------------+-------------+
No rows selected (0.049 seconds)
2、将程序jar包拷贝到slaves1的某个目录下
3、运行
[root@slaves1 bin]# ./spark-submit --class com.cn.util.KafkaEventProducer /home/test/telecomeAnalysis-1.0.0.jar
20/06/24 17:29:29 INFO producer.ProducerConfig: ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [192.168.230.21:6667, 192.168.230.22:6667, 192.168.230.23:6667]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
20/06/24 17:29:30 WARN producer.ProducerConfig: The configuration 'metadata.broker.list' was supplied but isn't a known config.
20/06/24 17:29:30 INFO utils.AppInfoParser: Kafka version : 1.0.2
20/06/24 17:29:30 INFO utils.AppInfoParser: Kafka commitId : 2a121f7b1d402825
{"user":"zhangSan","count_time":"2020-06-24 17:29:30","walk_place":"操场西北门"}
{"user":"liSi","count_time":"2020-06-24 17:29:35","walk_place":"操场东南北门"}
{"user":"wangWu","count_time":"2020-06-24 17:29:40","walk_place":"操场南门"}
{"user":"xiaoQiang","count_time":"2020-06-24 17:29:45","walk_place":"操场东门"}
^C[root@slaves1 bin]#