Spark Driver与Executor端添加调试信息

2019年4月10日 0 条评论 586 次阅读 0 人点赞

前言

由于Spark应用程序本身运行在JVM上运行，因此--verbose和--verbose:class选项都是可用的。--verbose会在输出设备上显示Java虚拟机运行的相关信息和Spark配置的详细信息，——verbose:class选项显示Driver和Executor加载的类。这些调试选项可以帮助用户在Driver和Executor端识别某些class类路径冲突。

适用环境

Spark 2.0, Apache Spark, Spark 2.1, Spark 2.x

配置方法

如果想要列出运行Java程序时JVM加载类的详细信息可以使用——verbose选项，该选项将输出一个列表且列表中包含了类加载器加载的所有类。使用——verbose:class选项的示例代码如下：

Shell
1
2
3
4
5
6
7
8
9
[your_account@36.36.36.36 spark-2.2.0-bin-hadoop2.6]$ bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--queue root.default \
--num-executors 20 \
--executor-memory 2048MB \
--conf "spark.driver.extraJavaOptions=-verbose:class" \
--conf "spark.executor.extraJavaOptions=-verbose:class"  \
--class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.2.0.jar 100000

登陆到YARN Web UI上，从Driver和Executor的stdout可以看到类加载的详细信息：

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor76 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor77 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedSerializationConstructorAccessor78 from __JVM_DefineClass__]
[Loaded scala.collection.parallel.immutable.ParIterable from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar]
[Loaded scala.collection.parallel.immutable.ParSeq from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar]
[Loaded scala.collection.parallel.immutable.ParVector from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar]
[Loaded org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$8$$anonfun$9 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$8$$anonfun$apply$10 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded io.netty.buffer.PoolArena$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/netty-all-4.0.43.Final.jar]
[Loaded scala.Product4$class from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/scala-library-2.11.8.jar]
[Loaded org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onExecutorMetricsUpdate$2$$anonfun$41 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$2 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.spark.executor.TaskMetrics$$anonfun$fromAccumulatorInfos$2$$anonfun$apply$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onExecutorMetricsUpdate$2$$anonfun$apply$17 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/spark-core_2.11-2.2.0.jar]
[Loaded org.apache.hadoop.hdfs.LeaseRenewer$2 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar]
[Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseRequestProto$Builder from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar]
[Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseResponseProto$Builder from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar]
[Loaded org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseResponseProto$1 from file:/data1/yarn/nm/usercache/your_account/filecache/10/__spark_libs__1451447860571528177.zip/hadoop-hdfs-2.6.5.jar]
[Loaded sun.reflect.GeneratedConstructorAccessor69 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedConstructorAccessor70 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor61 from __JVM_DefineClass__]

启动spark-shell或者使用spark-submit运行Spark程序时可以启用--verbose选项。--verbose选项添加以后将会输出细粒度的调试信息，例如Spark程序中各配置选项和相关class的加载位置。

Shell
1
[your_account@36.36.36.36 spark-2.2.0-bin-hadoop2.6]$ bin/spark-shell --verbose

命令行执行完毕后会打印出以下信息：

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
Using properties file: /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf
Adding default property: spark.port.maxRetries=1000
Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer
Adding default property: spark.sql.hive.convertMetastoreParquet=false
Adding default property: spark.history.fs.logDirectory=hdfs://demo/user/your_account/spark-2.2.0/history/
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.yarn.historyServer.address=36.36.36.36:18090
Adding default property: spark.history.fs.cleaner.enabled=true
Adding default property: spark.sql.hive.caseSensitiveInferenceMode=NEVER_INFER
Adding default property: spark.history.fs.cleaner.interval=1d
Adding default property: spark.history.fs.cleaner.maxAge=7d
Adding default property: spark.history.retainedApplications=200
Adding default property: spark.eventLog.dir=hdfs://demo/user/your_account/spark-2.2.0/history
Adding default property: spark.eventLog.compress=true
Adding default property: spark.history.ui.port=18090
Parsed arguments:
  master                  local[*]
  deployMode              null
  executorMemory          null
  executorCores           null
  totalExecutorCores      null
  propertiesFile          /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf
  driverMemory            null
  driverCores             null
  driverExtraClassPath    null
  driverExtraLibraryPath  null
  driverExtraJavaOptions  null
  supervise               false
  queue                   null
  numExecutors            null
  files                   null
  pyFiles                 null
  archives                null
  mainClass               org.apache.spark.repl.Main
  primaryResource         spark-shell
  name                    Spark shell
  childArgs               []
  jars                    null
  packages                null
  packagesExclusions      null
  repositories            null
  verbose                 true
 
Spark properties used, including those specified through
--conf and those from the properties file /home/your_path/spark/spark-2.2.0-bin-hadoop2.6/conf/spark-defaults.conf:
  (spark.history.fs.cleaner.enabled,true)
  (spark.yarn.historyServer.address,36.36.36.36:18090)
  (spark.eventLog.enabled,true)
  (spark.eventLog.compress,true)
  (spark.history.ui.port,18090)
  (spark.history.fs.cleaner.maxAge,7d)
  (spark.history.retainedApplications,200)
  (spark.serializer,org.apache.spark.serializer.KryoSerializer)
  (spark.history.fs.logDirectory,hdfs://demo/user/your_account/spark-2.2.0/history/)
  (spark.sql.hive.caseSensitiveInferenceMode,NEVER_INFER)
  (spark.eventLog.dir,hdfs://demo/user/your_account/spark-2.2.0/history)
  (spark.port.maxRetries,1000)
  (spark.sql.hive.convertMetastoreParquet,false)
  (spark.history.fs.cleaner.interval,1d)
 
 
Main class:
org.apache.spark.repl.Main
Arguments:
 
System properties:
(spark.history.fs.cleaner.enabled,true)
(spark.yarn.historyServer.address,36.36.36.36:18090)
(spark.eventLog.enabled,true)
(spark.eventLog.compress,true)
(spark.history.ui.port,18090)
(spark.history.fs.cleaner.maxAge,7d)
(SPARK_SUBMIT,true)
(spark.history.retainedApplications,200)
(spark.serializer,org.apache.spark.serializer.KryoSerializer)
(spark.history.fs.logDirectory,hdfs://demo/user/your_account/spark-2.2.0/history/)
(spark.app.name,Spark shell)
(spark.sql.hive.caseSensitiveInferenceMode,NEVER_INFER)
(spark.jars,)
(spark.submit.deployMode,client)
(spark.eventLog.dir,hdfs://demo/user/your_account/spark-2.2.0/history)
(spark.master,local[*])
(spark.port.maxRetries,1000)
(spark.sql.hive.convertMetastoreParquet,false)
(spark.history.fs.cleaner.interval,1d)
Classpath elements:
 
 
 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
21/11/30 15:47:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
Spark context Web UI available at http://36.36.36.36:4045
Spark context available as 'sc' (master = local[*], app id = local-1638258470147).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/
 
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112)
Type in expressions to have them evaluated.
Type :help for more information.

 

 

© 著作权归作者所有

勇敢，和生活的艰难无关。

文章评论（0）