目录
前言
由于Spark应用程序本身运行在JVM上运行，因此–verbose和–verbose:class选项都是可用的。–verbose会在输出设备上显示Java虚拟机运行的相关信息和Spark配置的详细信息，——verbose:class选项显示Driver和Executor加载的类。这些调试选项可以帮助用户在Driver和Executor端识别某些class类路径冲突。
适用环境
Spark 2.0, Apache Spark, Spark 2.1, Spark 2.x
配置方法
如果想要列出运行Java程序时JVM加载类的详细信息可以使用——verbose选项，该选项将输出一个列表且列表中包含了类加载器加载的所有类。使用——verbose:class选项的示例代码如下：
[your_account@36.36.36.36 spark-2.2.0-bin-hadoop2.6]$ bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --queue root.default \ --num-executors 20 \ --executor-memory 2048MB \ --conf "spark.driver.extraJavaOptions=-verbose:class" \ --conf "spark.executor.extraJavaOptions=-verbose:class" \ --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.2.0.jar 100000
登陆到YARN Web UI上，从Driver和Executor的stdout可以看到类加载的详细信息：
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor76 from __JVM_DefineClass__] [Loaded sun.reflect.GeneratedSerializationConstructorAccessor77 from __JVM_DefineClass__] [Loaded sun.reflect.GeneratedSerializationConstructorAccessor78 from __JVM_DefineClass__] [Loaded scala.collection.parallel.immutable.ParIterable from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar] [Loaded scala.collection.parallel.immutable.ParSeq from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar] [Loaded scala.collection.parallel.immutable.ParVector from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar] [Loaded org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$8$$anonfun$9 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$8$$anonfun$apply$10 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded io.netty.buffer.PoolArena$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/netty-all-4.0.43.Final.jar] [Loaded scala.Product4$class from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar] [Loaded org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onExecutorMetricsUpdate$2$$anonfun$41 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$2 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$2$$anonfun$apply$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onExecutorMetricsUpdate$2$$anonfun$apply$17 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar] [Loaded org.apache.hadoop.hdfs.LeaseRenewer$2 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar] [Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseRequestProto$Builder from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar] [Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseResponseProto$Builder from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar] [Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseResponseProto$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar] [Loaded sun.reflect.GeneratedConstructorAccessor69 from __JVM_DefineClass__] [Loaded sun.reflect.GeneratedConstructorAccessor70 from __JVM_DefineClass__] [Loaded sun.reflect.GeneratedMethodAccessor61 from __JVM_DefineClass__]
启动spark-shell或者使用spark-submit运行Spark程序时可以启用–verbose选项。–verbose选项添加以后将会输出细粒度的调试信息，例如Spark程序中各配置选项和相关class的加载位置。
[your_account@36.36.36.36 spark-2.2.0-bin-hadoop2.6]$ bin/spark-shell --verbose
命令行执行完毕后会打印出以下信息：
Using properties file: /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf
Adding default property: spark.port.maxRetries=1000
Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer
Adding default property: spark.sql.hive.convertMetastoreParquet=false
Adding default property: spark.history.fs.logDirectory=hdfs://demo/user/your_account/spark-2.2.0/history/
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.yarn.historyServer.address=36.36.36.36:18090
Adding default property: spark.history.fs.cleaner.enabled=true
Adding default property: spark.sql.hive.caseSensitiveInferenceMode=NEVER_INFER
Adding default property: spark.history.fs.cleaner.interval=1d
Adding default property: spark.history.fs.cleaner.maxAge=7d
Adding default property: spark.history.retainedApplications=200
Adding default property: spark.eventLog.dir=hdfs://demo/user/your_account/spark-2.2.0/history
Adding default property: spark.eventLog.compress=true
Adding default property: spark.history.ui.port=18090
Parsed arguments:
master local[*]
deployMode null
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf
driverMemory null
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass org.apache.spark.repl.Main
primaryResource spark-shell
name Spark shell
childArgs []
jars null
packages null
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf:
(spark.history.fs.cleaner.enabled,true)
(spark.yarn.historyServer.address,36.36.36.36:18090)
(spark.eventLog.enabled,true)
(spark.eventLog.compress,true)
(spark.history.ui.port,18090)
(spark.history.fs.cleaner.maxAge,7d)
(spark.history.retainedApplications,200)
(spark.serializer,org.apache.spark.serializer.KryoSerializer)
(spark.history.fs.logDirectory,hdfs://demo/user/your_account/spark-2.2.0/history/)
(spark.sql.hive.caseSensitiveInferenceMode,NEVER_INFER)
(spark.eventLog.dir,hdfs://demo/user/your_account/spark-2.2.0/history)
(spark.port.maxRetries,1000)
(spark.sql.hive.convertMetastoreParquet,false)
(spark.history.fs.cleaner.interval,1d)
Main class:
org.apache.spark.repl.Main
Arguments:
System properties:
(spark.history.fs.cleaner.enabled,true)
(spark.yarn.historyServer.address,36.36.36.36:18090)
(spark.eventLog.enabled,true)
(spark.eventLog.compress,true)
(spark.history.ui.port,18090)
(spark.history.fs.cleaner.maxAge,7d)
(SPARK_SUBMIT,true)
(spark.history.retainedApplications,200)
(spark.serializer,org.apache.spark.serializer.KryoSerializer)
(spark.history.fs.logDirectory,hdfs://demo/user/your_account/spark-2.2.0/history/)
(spark.app.name,Spark shell)
(spark.sql.hive.caseSensitiveInferenceMode,NEVER_INFER)
(spark.jars,)
(spark.submit.deployMode,client)
(spark.eventLog.dir,hdfs://demo/user/your_account/spark-2.2.0/history)
(spark.master,local[*])
(spark.port.maxRetries,1000)
(spark.sql.hive.convertMetastoreParquet,false)
(spark.history.fs.cleaner.interval,1d)
Classpath elements:
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
Spark context Web UI available at http://36.36.36.36:4045
Spark context available as 'sc' (master = local[*], app id = local-1638258470147).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112)
Type in expressions to have them evaluated.
Type :help for more information.
转载请注明：雪后西塘 » Spark Driver与Executor端添加调试信息