既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!
由于文件比较多,这里只是将部分目录截图出来,全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频,并且后续会持续更新
cp log4j2.xml.template log4j2.xml
cp celeborn-defaults.conf.template cp celeborn-defaults.conf
#### 3.1修改**celeborn-env.sh**
CELEBORN_MASTER_MEMORY=2g
CELEBORN_WORKER_MEMORY=2g
CELEBORN_WORKER_OFFHEAP_MEMORY=4g
#### 3.2 修改celeborn-defaults.conf
used by client and worker to connect to master
celeborn.master.endpoints 10.67.78.xx:9097
used by master to bootstrap
celeborn.master.host 10.67.78.xx
celeborn.master.port 9097
celeborn.metrics.enabled true
celeborn.worker.flusher.buffer.size 256k
If Celeborn workers have local disks and HDFS. Following configs should be added.
If Celeborn workers have local disks, use following config.
Disk type is HDD by defaut.
#celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=SSD
If Celeborn workers don’t have local disks. You can use HDFS.
Do not set celeborn.worker.storage.dirs
and use following configs.
celeborn.storage.activeTypes HDFS
celeborn.worker.sortPartition.threads 64
celeborn.worker.commitFiles.timeout 240s
celeborn.worker.commitFiles.threads 128
celeborn.master.slot.assign.policy roundrobin
celeborn.rpc.askTimeout 240s
celeborn.worker.flusher.hdfs.buffer.size 4m
celeborn.storage.hdfs.dir hdfs://10.67.78.xx:8020/celeborn
celeborn.worker.replicate.fastFail.duration 240s
If your hosts have disk raid or use lvm, set celeborn.worker.monitor.disk.enabled to false
celeborn.worker.monitor.disk.enabled false
### 4.复制到其他节点
scp -r /root/apache-celeborn-0.3.2-incubating-bin 10.67.78.xx1:/root/
scp -r /root/apache-celeborn-0.3.2-incubating-bin 10.67.78.xx2:/root/
因为在配置文件中已经配置了master 所以启动matster和worker即可。
### **5.启动master和worker**
cd $CELEBORN_HOME
./sbin/start-master.sh
./sbin/start-worker.sh celeborn://:
**之后在master的日志中看woker是否注册上**
![](https://img-blog.csdnimg.cn/direct/413b6199cb1a4bf29e0d9b53929562bb.png)
### 6.在 spark客户端使用
**复制 $CELEBORN\_HOME/spark/\*.jar 到 $SPARK\_HOME/jars/**
**修改spark-defaults.conf**
Shuffle manager class name changed in 0.3.0:
before 0.3.0: org.apache.spark.shuffle.celeborn.RssShuffleManager
since 0.3.0: org.apache.spark.shuffle.celeborn.SparkShuffleManager
spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
must use kryo serializer because java serializer do not support relocation
spark.serializer org.apache.spark.serializer.KryoSerializer
celeborn master
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
This is not necessary if your Spark external shuffle service is Spark 3.1 or newer
spark.shuffle.service.enabled false
options: hash, sort
Hash shuffle writer use (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
Sort shuffle writer uses less memory than hash shuffle writer, if your shuffle partition count is large, try to use sort hash writer.
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
dn.net/topics/618545628)**
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!