Spark提交任务依赖jar包与集群jar包冲突问题

趣说大数据 2021-03-05

3639

背景

在近期使用到google的guava包，本地程序使用的是guava-22.0.jar进行的开发和测试，在打包提交到集群运行时出现如下异常NoSuchMethodError,尝试了各种打包方法该异常仍然存在,在查阅资料后问题得到解决，在此记录一下。

Caused by: java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.from(Lcom/google/common/cache/CacheBuilderSpec;)Lcom/google/common/cache/CacheBuilder;

问题定位

--driver-java-options -verbose:class \

将程序加载调用的日志打到文件，在文件中进行查找

定位到该任务提交后加载的guava包版本为

[Loaded com.google.common.cache.CacheBuilder from file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/jars/guava-11.0.2.jar]

将对应的jar包下载到本地，使用gd-gui工具反编译打开CacheBuilder类发现该类中没有异常提示的方法。

问题解决

增加如下参数：

--conf "spark.driver.userClassPathFirst=true" \
--conf "spark.executor.userClassPathFirst=true" \

设置任务执行的时候使用用户提交程序所指定的jar包覆盖spark环境变量下的jar包即可解决问题。

官网说明

http://spark.apache.org/docs/latest/configuration.html#runtime-environment

spark.driver.userClassPathFirst 默认值false

   (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only.

spark.executor.userClassPathFirst 默认值false

(Experimental) Same functionality as spark.driver.userClassPathFirst, but applied to executor instances.

完整的提交脚本如下：

spark-submit \
--class com.xxxx.spark.xxxx.xxxx \
--deploy-mode client \
--driver-memory 2g \
--executor-memory 2g \
--executor-cores 2 \
--num-executors 4 \
--master local \
--conf "spark.driver.userClassPathFirst=true" \
--driver-java-options -verbose:class \
--jars /root/xxx/yy/xxx-1.5.14.jar,/root/xxx/yy/xxx-1.5.14.jar,/root/xxx/yy/fastjson-1.2.47.jar,/root/xxx/yy/guava-22.0.jar \
/root/xxx/yy/xxxx-3.0.jar 2021-02-27 ;

扫描二维码获取

Spark提交任务依赖jar包与集群jar包冲突问题

评论