服务器突然停电重启后 HBase起不来的问题

68 阅读 0 评论 45 点赞

我是靠谱客的博主冷傲香水，最近开发中收集的这篇文章主要介绍服务器突然停电重启后 HBase起不来的问题，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

错误日志信息

2013-10-24 14:51:22,436 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over -ROOT- starting at key ''
2013-10-24 14:51:22,436 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2013-10-24 14:51:22,440 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@19123eb0; serverName=hadoop5,60021,1382591816165
2013-10-24 14:51:22,457 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
2013-10-24 14:51:22,457 INFO org.apache.hadoop.hbase.catalog.MetaMigrationRemovingHTD: Meta version=0; migrated=true
2013-10-24 14:51:22,457 INFO org.apache.hadoop.hbase.catalog.MetaMigrationRemovingHTD: ROOT/Meta already up-to date with new HRI.
2013-10-24 14:51:22,464 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@19123eb0; serverName=hadoop5,60021,1382591816165
2013-10-24 14:51:22,474 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is hadoop3:60021
2013-10-24 14:51:22,475 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
2013-10-24 14:51:22,475 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2013-10-24 14:51:23,146 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2013-10-24 14:51:23,216 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2013-10-24 14:51:23,216 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.lang.NullPointerException
        at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2190)
        at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:323)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:501)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:326)
        at java.lang.Thread.run(Thread.java:662)
2013-10-24 14:51:23,218 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2013-10-24 14:51:23,218 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2013-10-24 14:51:23,218 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2013-10-24 14:51:23,218 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2013-10-24 14:51:23,218 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting

http://mail-archives.apache.org/mod_mbox/hbase-user/201203.mbox/%3CCAOcnVr1PpY0K6sPW6yzVyaL1ByuundVShFAeUGUinipYFbGg2g@mail.gmail.com%3E

这个网站有解释

Your problem isn't coprocessor related, per those logs. It appears
that one of the regions Master tried to look up in your .META. entries
had no region info associated with it (and hence got a null). Before
upgrading, did your hbck report proper health? Does your master not
start at all due to this?

参考网站http://blog.csdn.net/kirayuan/article/details/7170306

要明白这个问题的原因先弄清两个概念

-ROOT- && .META. Table

HBase中有两张特殊的Table，-ROOT-和.META.

Ø .META.：记录了用户表的Region信息，.META.可以有多个regoin

Ø -ROOT-：记录了.META.表的Region信息，-ROOT-只有一个region

Ø Zookeeper中记录了-ROOT-表的location

Client访问用户数据之前需要首先访问zookeeper，然后访问-ROOT-表，接着访问.META.表，最后才能找到用户数据的位置去访问，中间需要多次网络操作，不过client端会做cache缓存。

-ROOT- .META. 的位置在hdfs上 /hbase下面一般

这样当突然停电的时候可能region做分裂的时候失败了这样meta就不对了，要解决此问题需要修复meta表这也是hbase的单点故障。

今天遇到了这个问题最后的解决方案是重装了hbase，重建了hbase里面的表。幸好是实验环境，影响不大。

假如配置中存在Zookeeper，Client访问用户数据之前需要首先访问zookeeper，一旦ROOT区域被找到以后，Client就可以通过扫描ROOT区域找到相应的META区域去定位实际提供数据的HReginServer。

当定位到提供数据的HReginServer以后，Client就可以通过这个HReginServer找到需要的数据了。

这些信息将会被Client缓存起来，当下次请求的时候，就不需要重复查找。

当这些区域中的某个区域不可用的时候，Client将会逆向执行上面的过程，直到找到实际提供数据的HReginServer为止。

Table & Region

当Table随着记录数不断增加而变大后，会逐渐分裂成多份splits，成为regions，一个region由[startkey,endkey)表示，不同的region会被Master分配给相应的RegionServer进行管理：

Client

1 包含访问hbase的接口，client维护着一些cache来加快对hbase的访问，比如regione的位置信息。

Zookeeper

1 保证任何时候，集群中只有一个master

2 存贮所有Region的寻址入口。

3 实时监控Region Server的状态，将Region server的上线和下线信息实时通知给Master

4 存储Hbase的schema,包括有哪些table，每个table有哪些column family

Master

1 为Region server分配region

2 负责region server的负载均衡

3 发现失效的region server并重新分配其上的region

4 GFS上的垃圾文件回收

5 处理schema更新请求

Region Server

1 Region server维护Master分配给它的region，处理对这些region的IO请求

2 Region server负责切分在运行过程中变得过大的region

可以看到，client访问hbase上数据的过程并不需要master参与（寻址访问zookeeper和region server，数据读写访问regione server），master仅仅维护着table和region的元数据信息，负载很低。

HBase has two special tables that most users will never have to touch, “-ROOT-” and “.META.”. .META. holds the location of the regions of all the tables. -ROOT- then holds the location of .META. And because the regionserver holding -ROOT- can crash we won’t always know where -ROOT- lives, so we store the location of -ROOT- in a znode in a ZooKeeper.

In the original BigTable paper, .META. could span multiple regions, allowing a very large cluster. However, in practice HBase cluster sizes never grow beyond one region’s worth of meta information, so the .META. region never splits. This means the -ROOT- region is really just one row pointing to .META.

参考网址 http://blog.safaribooksonline.com/2012/11/16/tip-2-hbase-meta-layout/