jmx监控spark比storm稍微有点繁琐:
方法一、
首先在spark-defaults.conf中添加 ,但是8711端口不能重复,也就是说不能在一个节点上启动两个executor,或者端口冲突,没有storm友好
spark.executor.extraJavaOptions -XX:+PrintGCDetails
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8711
然后配置metrics.properties:,我只对executor监控,你可以替换*
executor.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
方法二:(好像有问题,只能监控SparkSubmit进程,不能监控CoarseGrainedExecutorBackend进程)
或者直接在spark-class文件配置,当是每次启动driver时候收到修改端口号,当然前提master和work已经启动了,不然启动会报端口错误
if [ -n "$SPARK_SUBMIT_BOOTSTRAP_DRIVER" ]; then
# This is used only if the properties file actually contains these special configs
# Export the environment variables needed by SparkSubmitDriverBootstrapper
echo "hello111111111111111111111111111111111111"
export RUNNER
export CLASSPATH
export JAVA_OPTS
#export JAVA_OPTS="-XX:MaxPermSize=128m $OUR_JAVA_OPTS -Dcom.sun.management.jmxremote.port=8300 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
export OUR_JAVA_MEM
export SPARK_CLASS=1
shift # Ignore main class (org.apache.spark.deploy.SparkSubmit) and use our own
exec "$RUNNER" org.apache.spark.deploy.SparkSubmitDriverBootstrapper "$@"
else
<span style="color:#FF6666;">export JAVA_OPTS="-XX:MaxPermSize=128m $OUR_JAVA_OPTS -Dcom.sun.management.jmxremote.port=8300
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
</span>
# Note: The format of this command is closely echoed in SparkSubmitDriverBootstrapper.scala
if [ -n "$SPARK_PRINT_LAUNCH_COMMAND" ]; then
echo -n "Spark Command: " 1>&2
echo "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@" 1>&2
echo -e "========================================\n" 1>&2
fi
exec "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@"
fi