先自我介绍一下,小编浙江大学毕业,去过华为、字节跳动等大厂,目前阿里P7
深知大多数程序员,想要提升技能,往往是自己摸索成长,但自己不成体系的自学效果低效又漫长,而且极易碰到天花板技术停滞不前!
因此收集整理了一份《2024年最新大数据全套学习资料》,初衷也很简单,就是希望能够帮助到想自学提升又不知道该从何学起的朋友。
既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!
由于文件比较多,这里只是将部分目录截图出来,全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频,并且后续会持续更新
如果你需要这些资料,可以添加V获取:vip204888 (备注大数据)
正文
docker rm -f kafka
docker run -d
–name kafka
–restart=always
–net host
–user 5000
–add-host=logaudit_kafka_01:192.168.11.100
–add-host=logaudit_kafka_02:192.168.11.101
–add-host=logaudit_kafka_03:192.168.11.102
-e KAFKA_NODE_ID=1
-e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://logaudit_kafka_01:9092
-e KAFKA_DAEMON_USER=5000
-e KAFKA_DAEMON_GROUP=5000
-e KAFKA_HEAP_OPTS=“-Xmx512m -Xms512m”
-e KAFKA_CFG_PROCESS_ROLES=broker,controller
-e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
-e KAFKA_CFG_SASL_MECHANISM_CONTROLLER_PROTOCOL=PLAIN
-e KAFKA_CONTROLLER_USER=contr0ller
-e KAFKA_CONTROLLER_PASSWORD=Contr0ller#XXXX
-e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
-e KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
-e KAFKA_ENABLE_KRAFT=yes
-e KAFKA_KRAFT_CLUSTER_ID=“Aqvf7RVETX-DInZbNUXXXXXXX”
-e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@192.168.11.100:9093,2@192.168.11.101:9093,3@192.168.11.102:9093
-e ALLOW_PLAINTEXT_LISTENER=yes
-e KAFKA_TLS_CLIENT_AUTH=none
-e KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN
-e KAFKA_CLIENT_LISTENER_NAME=PLAINTEXT
-e KAFKA_CLIENT_USERS=gohangout
-e KAFKA_CLIENT_PASSWORDS=Gohangout#XXXX
-e KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
-e KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=3
-e KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
-e KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=2
-e KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true
-e KAFKA_CFG_SUPER_USERS=User:gohangout
-e KAFKA_CFG_AUTHORIZER_CLASS_NAME=org.apache.kafka.metadata.authorizer.StandardAuthorizer
-v pwd
/etc/sasl_config.properties:/opt/bitnami/kafka/config/sasl_config.properties
-v pwd
/data:/bitnami/kafka/
-v /etc/localtime:/etc/localtime
bitnami/kafka:3.7.0
EOF
bash ${path}/start.sh
四、 kafka测试
docker exec -it kafka bash
#创建topic
kafka-topics.sh --create --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --topic test --command-config /opt/bitnami/kafka/config/sasl_config.properties
#生产
kafka-console-producer.sh --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --topic test --producer.config /opt/bitnami/kafka/config/sasl_config.properties
#消费
kafka-console-consumer.sh --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --topic test --consumer.config /opt/bitnami/kafka/config/sasl_config.properties
#扩容分区
kafka-topics.sh --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --alter --topic log-smartgate --partitions 3 --replication-factor 1 --command-config /opt/bitnami/kafka/config/sasl_config.properties
#查询分区
kafka-topics.sh --describe --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --topic test --command-config /opt/bitnami/kafka/config/sasl_config.properties
#acl ,需要在添加启动参数
kafka-acls.sh --bootstrap-server 192.168.11.100:9092,192.168.11.101:9092,192.168.11.102:9092 --add --allow-principal User:gohangout --operation ALL --topic test --command-config /opt/bitnami/kafka/config/sasl_config.properties
Adding ACLs for resource ResourcePattern(resourceType=TOPIC, name=test, patternType=LITERAL)
:
(principal=User:gohangout, host=*, operation=ALL, permissionType=ALLOW)
Current ACLs for resource ResourcePattern(resourceType=TOPIC, name=test, patternType=LITERAL)
:
(principal=User:gohangout, host=*, operation=ALL, permissionType=ALLOW)
#这三台配置是与ALC有关
#-e KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true
#-e KAFKA_CFG_SUPER_USERS=User:gohangout
#-e KAFKA_CFG_AUTHORIZER_CLASS_NAME=org.apache.kafka.metadata.authorizer.StandardAuthorizer \
#ACL有以下权限
Describe
DescribeConfigs
Alter
IdempotentWrite
Read
Delete
Create
ClusterAction
All
CreateTokens
DescribeTokens
Write
AlterConfigs
#查看ACL授权
kafka-acls.sh --bootstrap-server 192.168.11.103:9092 --list --command-config /opt/bitnami/kafka/config/sasl_config.properties
Current ACLs for resource ResourcePattern(resourceType=TOPIC, name=test, patternType=LITERAL)
:
(principal=User:gohangout, host=*, operation=ALL, permissionType=ALLOW)
四、安装redpanda
path=/data/kowl
mkdir -p ${path}/etc
cat > ${path}/etc/console.yaml << ‘EOF’
server:
listenAddress:
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
需要这份系统化的资料的朋友,可以添加V获取:vip204888 (备注大数据)
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
数据)**
[外链图片转存中…(img-zGREHQPh-1713304819204)]
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!