hdfs datanode 启动失败

68 阅读 0 评论 45 点赞

我是靠谱客的博主单纯火，最近开发中收集的这篇文章主要介绍hdfs datanode 启动失败，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

hadoop-root-datanode-ubuntu.log中：

2015-03-12 23:52:33,671 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000. Exiting.
java.io.IOException: Incompatible clusterIDs in /hdfs/name/dfs/data: namenode clusterID = CID-70d64aad-1dfe-4f87-af15-d53ff80db3dd; datanode clusterID = CID-388a9ec6-cb87-4b0d-97c4-3b4d5c787b76
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:646)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:320)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:403)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:422)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1311)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1276)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:314)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:828)
        at java.lang.Thread.run(Thread.java:745)
2015-03-12 23:52:33,680 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000
2015-03-12 23:52:33,788 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2015-03-12 23:52:35,790 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2015-03-12 23:52:35,791 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2015-03-12 23:52:35,792 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ubuntu/127.0.1.1
************************************************************/

原因是：

namenode与datanode的clusterID在重新格式化namenode以后已经不再匹配，datanode无法启动。

另外：

此错误会导致在hive导入数据时发生如下错误（由于metadata不存在hdfs中，故create table并无报错）：

hive> load data local inpath '/root/dbfile' overwrite into table employees PARTITION (country='US', state='IL');
Loading data to table default.employees partition (country=US, state=IL)
Failed with exception Unable to move source file:/root/dbfile to destination hdfs://localhost:9000/user/hive/warehouse/employees/country=US/state=IL/dbfile
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTas

解决方法：

将hdfs存储数据的所在目录删掉，重新格式化hdfs（相关参数： dfs.name.dir dfs.data.dir）：

hadoop namenode -format

转载于:https://www.cnblogs.com/JingJ/p/4336926.html