Ubuntu 14.04.1 LTS 安装Hadoop 2.7.0(单节点) 一

56 阅读 0 评论 37 点赞

我是靠谱客的博主优雅心锁，最近开发中收集的这篇文章主要介绍Ubuntu 14.04.1 LTS 安装Hadoop 2.7.0(单节点) 一，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

自己想玩hadoop，但总是由于种种异常导致不想玩了。今天呢天气太热了，没地方玩只能在家搞hadoop了，不过还好最终能启动成功了。以此来记录下过程方便以后自己参考，也给想玩hadoop的程序员提供一个参考

1.下载解压就不说了，我下载的是2.7.0版本的。

2.hadoop解压目录/opt/app/hadoop/hadoop-2.7.0。

在/opt/app/hadoop/目录下创建一下目录

tmp、hdfs、hdfs/data、hdfs/name

3.配置/etc/hadoop目录下的core-site.xml

<configuration>
  <property>
      <name>fs.defaultFS</name>
      <value>hdfs://192.168.56.101:9000</value>
  </property>
  <property>
      <name>hadoop.tmp.dir</name>
      <value>file:/home/hadoop/tmp</value>
  </property>
  <property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
</configuration>

4.配置/etc/hadoop目录下的hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/app/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/app/hadoop/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>192.168.56.101:9001</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
</configuration>

5.配置/etc/hadoop目录下mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
         <property>
                <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>192.168.56.101:8032</value>
        </property>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>192.168.56.101:8030</value>
        </property>
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>192.168.56.101:8031</value>
        </property>
        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>192.168.56.101:8033</value>
        </property>
        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>192.168.56.101:8088</value>
        </property>
        <property>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>1024</value>
        </property>
        <property>
                 <name>mapreduce.jobhistory.address</name>
                 <value>192.168.56.101:10020</value>
        </property>
        <property>
                 <name>mapreduce.jobhistory.webapp.address</name>
                 <value>192.168.56.101:19888</value>
        </property>
</configuration>

6.配置/etc/hadoop目录下etc/hadoop/yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

7.配置/etc/hadoop目录下hadoop-env.sh、yarn-env.sh的JAVA_HOME

#Java setting
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_79

8.配置环境变量

在~/.profile文件中添加环境变量

vim ~/.profile

#setting java
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_79
export JRE_HOME=${JAVA_HOME}/jre

#setting hadoop
export HADOOP_HOME=/opt/app/hadoop/hadoop-2.7.0
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

不设置的话启动hadoop会报错。

错误如下：

hadoop@ubuntu:~$ /opt/app/hadoop/hadoop-2.7.0/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/01/23 20:23:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Java HotSpot(TM) Client VM warning: You have loaded library /hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
hd-m1]
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
hd-m1: starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-hd-m1.out
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
Client: ssh: Could not resolve hostname Client: Temporary failure in name resolution
You: ssh: Could not resolve hostname You: Temporary failure in name resolution
warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution......等等