大数据
蓝色的城墙
这个作者很懒,什么都没留下…
展开
-
SparkSession中的大坑
报错内容:Make sure that term plans is in your classpath and check for conflicting dependencies with -Ylog-classpath.Error:(52, 15) Symbol 'term org.apache.spark.sql.catalyst.plans' is missing from the classpath.This symbol is required by 'value org.apache.s原创 2021-02-23 15:23:03 · 1909 阅读 · 0 评论 -
Spark 开发过程中的io.netty冲突问题
io.netty冲突问题下面展示一些 内联代码片。Exception in thread "main" java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.defaultUseCacheForAllThreads()Z at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:120) at org原创 2021-02-21 21:57:00 · 1073 阅读 · 0 评论 -
Kafka开发中的错误
Kafka开法的错误1:kafka报错内容:WARN clients.NetworkClient: [Consumer clientId=consumer-1, groupId=console-consumer-26747] Connection to node -1 (bigdatacdh1/192.168.1.101:9092) could not be established. Broker may not be available.kafka 的服务可能没有开启,从而导致Broker may原创 2020-08-21 12:26:07 · 2266 阅读 · 0 评论 -
【工具类】单例+读取配置文件
【工具类】单例+读取配置文件配置文件kafka-server-config.properties#brokerbootstrap.servers = bigdatacdh1:9092,bigdatacdh2:9092,bigdatacdh3:9092#zk.connect = bigdatacdh1:2181,bigdatacdh2,bigdatacdh3:2181#key.serializer = org.apache.kafka.common.serialization.StringDeser原创 2020-08-20 00:05:02 · 138 阅读 · 0 评论 -
【大数据】CDH6.3.X安装
CDH6.3.X安装基本设置1.修改网卡信息cd /etc/sysconfig/network-scriptsvim ifcfg-enXXXXXXXa.修改BOOTPROTO=staticONBOOT=yesb.在结尾增加IPADDR=192.168.1.101GATEWAY=192.168.247.2NETMASK=255.255.255.0DNS1=8.8.8.8DNS2=114.114.114.114 修改其他2台主机IP分别为 192.1原创 2020-08-19 00:56:05 · 245 阅读 · 0 评论 -
【大数据】【kafka】kafka生产者和消费者测试
参考:http://dblab.xmu.edu.cn/blog/1096-2/前提条件:启动zookeeper,kafka.1.启动zookeeper;bin/zkServer.sh start2.启动Kafka;bin/kafka-server-start.sh config/server.properties总结:kafka的topic测试过程需要三步,1.创建topic ;bin/kafka-topics.sh --create --zookeeper localhost:2181 --re原创 2020-05-21 02:06:22 · 639 阅读 · 0 评论