Spark集群搭建时的踩得坑

• 启动时

192.168.5.176: failed to launch: nice -n 0 /usr/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://server178:7077
192.168.5.176:          at java.lang.Class.getMethod0(Class.java:3018)
192.168.5.176:          at java.lang.Class.getMethod(Class.java:1784)
192.168.5.176:          at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
192.168.5.176:          at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
192.168.5.176:   Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
192.168.5.176:          at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
192.168.5.176:          at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
192.168.5.176:          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
192.168.5.176:          at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
192.168.5.176:          ... 7 more
192.168.5.176: full log in /usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-server176.out
192.168.5.179: failed to launch: nice -n 0 /usr/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://server178:7077
192.168.5.179:          at java.lang.Class.getMethod0(Class.java:3018)
192.168.5.179:          at java.lang.Class.getMethod(Class.java:1784)
192.168.5.179:          at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
192.168.5.179:          at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
192.168.5.179:   Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
192.168.5.179:          at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
192.168.5.179:          at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
192.168.5.179:          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
192.168.5.179:          at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
192.168.5.179:          ... 7 more
192.168.5.179: full log in /usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-server179.out
192.168.5.178: failed to launch: nice -n 0 /usr/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://server178:7077
192.168.5.178:          at java.lang.Class.getMethod0(Class.java:3018)
192.168.5.178:          at java.lang.Class.getMethod(Class.java:1784)
192.168.5.178:          at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
192.168.5.178:          at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
192.168.5.178:   Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
192.168.5.178:          at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
192.168.5.178:          at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
192.168.5.178:          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
192.168.5.178:          at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
192.168.5.178:          ... 7 more
192.168.5.178: full log in /usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-server178.out

• worker日志

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 7 more

解决方法

• 在conf/spark-env.sh中添加

export SPARK_DIST_CLASSPATH=$(/usr/hadoop-2.9.2/bin/hadoop classpath)

• 或者 配置了hadoop的环境变量可以使用下面的配置

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

• 在中sbin/spark-config.sh中添加

export JAVA_HOME=/usr/local/java/jdk1.8.0_231