Hbase无法启动及web查看出现500错误的原因及解决方法

80 阅读 0 评论 53 点赞

我是靠谱客的博主欣喜帅哥，最近开发中收集的这篇文章主要介绍Hbase无法启动及web查看出现500错误的原因及解决方法，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

偶然一次停电，导致机房机器重启，于是很多机器静态IP冲突失效。好不容易把网络调整好，Hbase集群却无法正常启动，Hadoop却可以正常使用。大致情况如下：

正常启动Hadoop和zookeeper，

然后启动Hbase，jps查看一切正常。

当打开web页面（60010）查看的时候却出现500错误，错误大致如下：

HTTP ERROR: 500
Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1

RequestURI=/master.jsp

Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1

at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

查看Hbase master的日志，大致错误如下：

2011-12-08 14:34:54,305 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.

2011-12-08 14:34:54,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.

2011-12-08 14:34:55,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.

2011-12-08 14:34:55,307 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.10.11.184:60020

2011-12-08 14:34:55,308 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.

2011-12-08 14:35:15,404 INFO org.apache.hadoop.hbase.master.ServerManager: 4 region servers, 0 dead, average load 0.3333333333333333

2011-12-08 14:35:15,484 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.10.11.184:60020, regionname: -ROOT-,,0, startKey: <>}

2011-12-08 14:35:15,485 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.

2011-12-08 14:35:15,485 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1

at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:429)

at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:918)

at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:934)

at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:173)

at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)

at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)

at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)

at org.apache.hadoop.hbase.Chore.run(Chore.java:68)

日志呈现结果大致是无法连接到 /10.10.11.184:60020 ,不能连接到regionServer

于是查看regionServer的日志，初次一看感觉没什么错误，就没怎么注意，然后继续纠结在其他地方……后来多次失败后重新审视日志的时候才发现了一些端倪，如下

2011-12-08 22:45:02,365 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Created ZNode /hbase/rs/1323355502342 with data 0:0:0:0:0:0:0:1:60020

2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=10.10.11.184

2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://hadoop5-virtual-machine:9000/hbase

2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://hadoop5-virtual-machine:9000/hbase

2011-12-08 22:45:02,395 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address to use. Was=0:0:0:0:0:0:0:1:60020, Now=10.10.11.184

2011-12-08 22:45:02,510 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://hadoop5-virtual-machine:9000/hbase/.logs/hadoop4-virtual-machine,60020,1323355502342

2011-12-08 22:45:02,521 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog configuration: blocksize=67108864, rollsize=63753420, enabled=true, flushlogentries=100, optionallogflushinternal=10000ms

2011-12-08 22:45:02,561 INFO org.apache.hadoop.hbase.regionserver.HLog: New hlog /hbase/.logs/hadoop4-virtual-machine,60020,1323355502342/hlog.dat.1323355502522

2011-12-08 22:45:02,566 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=RegionServer, sessionId=regionserver/0:0:0:0:0:0:0:1:60020

然后我在slave节点下jps查看，发现hbase相关的regionServer的进程挂掉了，而master的进程还在。

由上蓝色标记字体看出：为什么会出现IP6的地址呢？检查配置文件也没有出错，因为所有的配置文件都是用主机名替代了IP地址。于是就想到了是不是hosts文件的映射出错了。

打开/etc/hosts文件，果不其然，regionServer的主机名映射的是 ::1，

我记得这个主机名和当前IP的映射应该是网卡初始化的时候由NetworkManager自动加上去的，

把 ::1改成 regionServer的实际地址(或重新添加实际地址与主机名的映射)，重启Hbase，访问WEB站点，成功启动！

回顾整个hadoop集群启动失败这个过程，先是断电，然后是静态IP失效，问题应该就是在这个时候产生的，在实效的同试hosts文件被修改。

其实HBASE集群配置相当简单，但由于一些小小的失误，往往是我们自以为不会出错的地方，查看log的时候也不细心，然后以至于浪费我们很多的时间。至于IP与主机名映射的问题是我们配置集群的时候最容易忽视的地方，大家在下次碰到这类相关问题的时候不妨去看看hosts文件。