java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration

62 阅读 0 评论 41 点赞

我是靠谱客的博主迷路往事，最近开发中收集的这篇文章主要介绍java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

用sbt打包Spark程序，并未将所有依赖都打入包中，把Spark应用放到集群中运行时，出现异常：

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at SparkHbase $.main(SparkHbase.scala:34) at SparkHbase.main(SparkHbase.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$ .org $apache$ spark $deploy$ SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
… 11 more
出现该异常的原因是Spark应用缺少hbase依赖，我这里的做法是在集群的spark/conf/spark-env.sh中添加下文：

export SPARK_CLASSPATH=/home/hadoop/SW/hbase/lib/hbase-client-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-server-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-common-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-protocol-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/htrace-core-2.04.jar:/home/hadoop/SW/hbase/lib/hbase-hadoop2-compat-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-it-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/guava-12.0.1.jar
切记注意每个jar包之间用冒号分隔！然后执行命令：

source spark-env.sh
并重启一下spark服务，就ok了！

其实还有一个方法，就是在你提交应用时增加–driver-class-path配置参数来设置driver的classpath：

./spark-submit –driver-class-path /home/hadoop/SW/hbase/lib/hbase-client-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-server-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-common-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-protocol-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/htrace-core-2.04.jar:/home/hadoop/SW/hbase/lib/hbase-hadoop2-compat-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/hbase-it-0.98.12-hadoop2.jar:/home/hadoop/SW/hbase/lib/guava-12.0.1.jar –class com.dtxy.data.SqlTest ../lib/bigdata-1.0-SNAPSHOT.jar
注：不能同时在spark/conf/spark-env.sh里面配置SPARK_CLASSPATH又在提交作业加上–driver-class-path参数，否则会出现异常：

15/08/14 09:22:23 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
at org.apache.spark.SparkConf

a n o n f u n $ v a l i d a t e S e t t i n g s $ 6

$anonfun$validateSettings$6$ anonfun

apply $apply$ 8.apply(SparkConf.scala:444)
at org.apache.spark.SparkConf

a n o n f u n $ v a l i d a t e S e t t i n g s $ 6

$anonfun$validateSettings$6$ anonfun

apply $apply$ 8.apply(SparkConf.scala:442)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.SparkConf

a n o n f u n $ v a l i d a t e S e t t i n g s $ 6. a p p l y (S p a r k C o n f . s c a l a : 442) a t o r g . a p a c h e . s p a r k . S p a r k C o n f

$anonfun$validateSettings$6.apply(SparkConf.scala:442) at org.apache.spark.SparkConf$ anonfun

validateSettings $validateSettings$ 6.apply(SparkConf.scala:430)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:430)
at org.apache.spark.SparkContext.(SparkContext.scala:365)
at com.dtxy.data.SqlTest

.main(SqlTest.scala:27)atcom.dtxy.data.SqlTest.main(SqlTest.scala)atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:606)atorg.apache.spark.deploy.SparkSubmit $.main(SqlTest.scala:27) at com.dtxy.data.SqlTest.main(SqlTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$ .org

apache $apache$ spark

deploy $deploy$ SparkSubmit

r u n M a i n (S p a r k S u b m i t . s c a l a : 664) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . d o R u n M a i n $ 1 (S p a r k S u b m i t . s c a l a : 169) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . s u b m i t (S p a r k S u b m i t . s c a l a : 192) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . m a i n (S p a r k S u b m i t . s c a l a : 111) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t . m a i n (S p a r k S u b m i t . s c a l a) 15 / 08 / 1409 : 22 : 23 I N F O S p a r k C o n t e x t : S u c c e s s f u l l y s t o p p e d S p a r k C o n t e x t E x c e p t i o n i n t h r e a d “ m a i n ” o r g . a p a c h e . s p a r k . S p a r k E x c e p t i o n : F o u n d b o t h s p a r k . d r i v e r . e x t r a C l a s s P a t h a n d S P A R K C L A S S P A T H . U s e o n l y t h e f o r m e r . a t o r g . a p a c h e . s p a r k . S p a r k C o n f

$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/08/14 09:22:23 INFO SparkContext: Successfully stopped SparkContext Exception in thread “main” org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former. at org.apache.spark.SparkConf$ anonfun

validateSettings $validateSettings$ 6

a n o n f u n $ a p p l y $ 8. a p p l y (S p a r k C o n f . s c a l a : 444) a t o r g . a p a c h e . s p a r k . S p a r k C o n f

$anonfun$apply$8.apply(SparkConf.scala:444) at org.apache.spark.SparkConf$ anonfun

validateSettings $validateSettings$ 6

a n o n f u n $ a p p l y $ 8. a p p l y (S p a r k C o n f . s c a l a : 442) a t s c a l a . c o l l e c t i o n . i m m u t a b l e . L i s t . f o r e a c h (L i s t . s c a l a : 318) a t o r g . a p a c h e . s p a r k . S p a r k C o n f

$anonfun$apply$8.apply(SparkConf.scala:442) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.SparkConf$ anonfun

validateSettings $validateSettings$ 6.apply(SparkConf.scala:442) at org.apache.spark.SparkConf

a n o n f u n $ v a l i d a t e S e t t i n g s $ 6. a p p l y (S p a r k C o n f . s c a l a : 430) a t s c a l a . O p t i o n . f o r e a c h (O p t i o n . s c a l a : 236) a t o r g . a p a c h e . s p a r k . S p a r k C o n f . v a l i d a t e S e t t i n g s (S p a r k C o n f . s c a l a : 430) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . (S p a r k C o n t e x t . s c a l a : 365) a t c o m . d t x y . d a t a . S q l T e s t $ . m a i n (S q l T e s t . s c a l a : 27) a t c o m . d t x y . d a t a . S q l T e s t . m a i n (S q l T e s t . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 57) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 606) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . o r g $ a p a c h e $ s p a r k $ d e p l o y $ S p a r k S u b m i t

$anonfun$validateSettings$6.apply(SparkConf.scala:430) at scala.Option.foreach(Option.scala:236) at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:430) at org.apache.spark.SparkContext.(SparkContext.scala:365) at com.dtxy.data.SqlTest$.main(SqlTest.scala:27) at com.dtxy.data.SqlTest.main(SqlTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$ runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit

.doRunMain $.doRunMain$ 1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit

.submit(SparkSubmit.scala:192)atorg.apache.spark.deploy.SparkSubmit $.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$ .main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/08/14 09:22:23 INFO Utils: Shutdown hook called
到此为止，问题解决！

参考来源：http://www.abcn.net/2014/07/lighting-spark-with-hbase-full-edition.html