w757227129的博客

威胁情报组收到事件情报，部分单位发现Nacos存在权限认证绕过的Nday漏洞（CVE-2021-29441），据在野验证，该漏洞在Nacos版本2.0.4及之前可复现。

开启mongodb副本集的脚本

mongodb开启副本，mongodb实时数据获取

spark 3.1.2版本对多版本hive的兼容处理

解决方式1:配置环境变量:PYTHONPATHexport PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.2-src.zip:$PYTHONPATH解决方式2:使用findspark模块import findsparkfindspark.init()

spark业务开发-列值替换项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据order_number,order_date,purchaser,quantity,product_id,remark10001,2016-01-16,1001,1,102,机q器w记e录r10003,2016-01-17,1002,2,105,人工记录10002,2016-01-19,1002,3,106,人工补录10004,2016-02-21,

spark业务开发-排序项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据subject,name,score数学,张三,88语文,张三,92英语,张三,77数学,王五,65语文,王五,87英语,王五,90数学,李雷,67语文,李雷,33英语,李雷,24数学,宫九,77语文,宫九,87英语,宫九,90输出数据+-------+----+-----+|subject|name|score|+------

spark业务开发-union合并(union)项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据集1id,name,profession,enroll,score1,庄劲聪,经济学类,北京理工大学,5512,吴雅思,经济学类,北京理工大学,5293,周育传,经济学类,北京理工大学,6824,丁俊伟,通信工程,北京电子科技学院,7085,庄逸琳,通信工程,北京电子科技学院,7086,吴志发,通信工程,北京电子科技学院,578

spark业务开发-列拆分项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据id,data1,"Ming,20,15552211521"2,"hong,19,13287994007"3,"zhi,21,15552211523"输出数据+---+----+---+-----------+| id| 列1|列2| 列3|+---+----+---+-----------+| 1|Ming| 20|15552

spark业务开发-添加索引列项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据name,profession,enroll,score曾凰妹,金融学,北京电子科技学院,637谢德炜,金融学,北京电子科技学院,542林逸翔,金融学,北京电子科技学院,543王丽云,金融学,北京电子科技学院,626吴鸿毅,金融学,北京电子科技学院,591施珊珊,经济学类,北京理工大学,581柯祥坤,经济学类,北京理工大学,650庄劲聪,经济

spark业务开发-行转列项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据subject,name,score数学,张三,88语文,张三,92英语,张三,77数学,王五,65语文,王五,87英语,王五,90数学,李雷,67语文,李雷,33英语,李雷,24数学,宫九,77语文,宫九,87英语,宫九,90输出数据+-------+----+----+----+----+|subject|宫九|张三|李雷|

spark业务开发-删除重复行项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据order_number,order_date,purchaser,quantity,product_id,remark10001,2016-01-16,1001,1,102,机q器w记e录r10003,2016-01-17,1002,2,105,人工记录10002,2016-01-19,1002,3,106,人工补录10004,2016-02-21

spark业务开发-列选择项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据"id","name","description","weight""102","car battery","12V car battery","8.1""103","12-pack drill bits","12-pack of drill bits with sizes ranging from #40 to #3","0.8""104","hamme

spark业务开发-列转行项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据subject,gongjiu,zhangsan,lilei,wangwu英语,90,77,24,90语文,87,92,33,87数学,77,88,67,65输出数据+-------+----+-----+|subject|name|score|+-------+----+-----+| 英语| 77| 77|| 英语| 90

spark业务开发-数据清洗项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据order_number,order_date,purchaser,quantity,product_id,remark10001,2016-01-16,1001,1,102,机q器w记e录r10003,2016-01-17,1002,2,105,人工记录10002,2016-01-19,1002,3,106,人工补录10004,2016-02-21,

spark业务开发-聚合(agg)项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据name,profession,enroll,score曾凰妹,金融学,北京电子科技学院,637谢德炜,金融学,北京电子科技学院,542林逸翔,金融学,北京电子科技学院,543王丽云,金融学,北京电子科技学院,626吴鸿毅,金融学,北京电子科技学院,591施珊珊,经济学类,北京理工大学,581柯祥坤,经济学类,北京理工大学,650庄劲聪,

spark业务开发-空值处理项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据集"id","name","description","weight""102","car battery","12V car battery","8.1""103","12-pack drill bits","12-pack of drill bits with sizes ranging from #40 to #3","0.8""104","ham

spark业务开发-join合并(join)项目地址:https://gitee.com/cch-bigdata/spark-process.git输入数据集1order_number,order_date,purchaser,quantity,product_id,remark10001,2016-01-16,1001,1,102,机q器w记e录r10003,2016-01-17,1002,2,105,人工记录10002,2016-01-19,1002,3,106,人工补录10004,2

spark业务开发-列过滤(filter)输入数据order_number,order_date,purchaser,quantity,product_id,remark10001,2016-01-16,1001,1,102,机q器w记e录r10003,2016-01-17,1002,2,105,人工记录10002,2016-01-19,1002,3,106,人工补录10004,2016-02-21,1003,4,107,自然交易10001,2016-01-16,1001,1,102,机器记录

CDC框架之Debezium简介Debezium 是一组分布式服务，用于捕获数据库中的更改，以便您的应用程序可以看到这些更改并对其做出响应。Debezium 在更改事件流中记录每个数据库表中的所有行级更改，应用程序只需读取这些流以查看更改事件发生的相同顺序。Debezium 构建在Apache Kafka之上，并提供与Kafka Connect兼容的连接器，用于监控特定的数据库管理系统。Debezium 在 Kafka 日志中记录数据更改的历史记录，从您的应用程序使用它们的位置。这使您的应用程序可以轻

kudu集群安装服务器masterslave01slave02架构master:kudu-master,kudu-tserverslave01:kudu-tserverslave02:kudu-tserver前置条件安装ntp同步时间，否则会报错误Check failed: _s.ok() Bad status:Service unavailable: Cannot initialize clock: Error reading clock. Clock con

Hadoop Shell命令cat使用方法：hadoop fs -cat URI [URI …]将路径指定文件的内容输出到stdout。示例：hadoop fs -cat hdfs://host1:port1/file1 hdfs://host2:port2/file2hadoop fs -cat file:///file3 /user/hadoop/file4返回值：成功返回0，失败返回-1。chgrp使用方法：hadoop fs -chgrp [-R] GROUP URI [UR

1. 创建自定义docker networkdocker network create --subnet=172.20.0.0/16 hadoop-cluster2. 创建自定义镜像(基于centos) 拉取centos镜像:docker pullcentos:latest 运行容器：docker run -d --privileged --name cluster-master -h cluster-master --net hadoop-n...