HA HADOOP集群和HIVE部署

1.集群环境节点分布

2.修改主机名

3.设置免密码登录

4.设置环境变量

5.搭建hadoop集群

5.1 修改配置文件

5.1.1修改core-stie.xml

	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://bigdatasvr02:9000</value>
	</property>
	<property>
		<name>io.file.buffer.size</name>
		<value>131072</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>file:/home/hadoop/hadoop/tmp</value>
		<description>Abasefor other temporary directories.</description>
	</property>

5.1.2修改hdfs-site.xml

	 <property>
		<name>dfs.nameservices</name>
		<value>hadoopcluster</value>
	</property>
	
	<property>
		<name>dfs.ha.namenodes.hadoopcluster</name>
		<value>nn1,nn2</value>
	</property>

	<property>
		<name>dfs.namenode.rpc-address.hadoopcluster.nn1</name>
		<value>bigdatasvr02:9000</value>
	</property>
	<property>
		<name>dfs.namenode.rpc-address.hadoopcluster.nn2</name>
		<value>bigdatasvr03:9000</value>
	</property>
	<property>
		<name>dfs.namenode.http-address.hadoopcluster.nn1</name>
		<value>bigdatasvr02:50070</value>
	</property>
	<property>
		<name>dfs.namenode.http-address.hadoopcluster.nn2</name>
		<value>bigdatasvr03:50070</value>
	</property>
	
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/home/hadoop/hadoop/ha/hdfs/name</value>
		<description>allow multiple directory split by ,</description>
	</property>
	<property>
		<name>dfs.namenode.shared.edits.dir</name>
		<value>qjournal://bigdatasvr01:8485;bigdatasvr02:8485;bigdatasvr03:8485/hadoopcluster</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/home/hadoop/hadoop/ha/hdfs/data</value>
		<description>allow multiple directory split by ,</description>
	</property>

	<property>
		<name>dfs.ha.automatic-failover.enabled</name>
		<value>false</value>
	<description>Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA configuration.</description>
	</property>
	<property>
		<name>dfs.journalnode.edits.dir</name>
		<value>/home/hadoop/hadoop/ha/hdfs/journal</value>
	</property>
	
	 <property>
	  <name>dfs.replication</name>
	  <value>2</value>
	 </property>

	<property>
		 <name>dfs.webhdfs.enabled</name>
	     <value>true</value>
	 </property>

5.1.3修改mapred-site.xml

 <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
  
 <property>
	<name>mapreduce.jobhistory.address</name>
	<value>bigdatasvr03:10020</value>
 </property>
 <property>
	<name>mapreduce.jobhistory.webapp.address</name>
	<value>bigdatasvr03:19888</value>
 </property>

5.1.4修改yarn-site.xml

	<property>
		<description>The hostname of the RM.</description>
		<name>yarn.resourcemanager.hostname</name>
		<value>bigdatasvr02</value>
	</property>    
	<property>
		<description>The address of the applications manager interface in the RM.</description>
		<name>yarn.resourcemanager.address</name>
		<value>${yarn.resourcemanager.hostname}:8032</value>
	</property>
	<property>
		<description>The http address of the RM web application.</description>
		<name>yarn.resourcemanager.webapp.address</name>
		<value>${yarn.resourcemanager.hostname}:8088</value>
	</property>
	<property>
		<description>The https adddress of the RM web application.</description>
		<name>yarn.resourcemanager.webapp.https.address</name>
		<value>${yarn.resourcemanager.hostname}:8090</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address</name>
		<value>${yarn.resourcemanager.hostname}:8031</value>
	</property>
	<property>
	   <name>yarn.resourcemanager.scheduler.address</name>
	   <value>${yarn.resourcemanager.hostname}:8030</value>
	</property>
	<property>
		<description>The address of the RM admin interface.</description>
		<name>yarn.resourcemanager.admin.address</name>
		<value>${yarn.resourcemanager.hostname}:8033</value>
	</property>
	<property>
		<description>List of directories to store localized files in. An application's localized file directory will be found in:
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
Individual containers' work directories, called container_${contid}, will
be subdirectories of this.</description>
		<name>yarn.nodemanager.local-dirs</name>
		<value>/home/hadoop/hadoop/ha/yarn/local</value>
	</property>
	<property>
		<description>Whether to enable log aggregation</description>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
	</property>
	<property>
		<description>Where to aggregate logs to.</description>
		<name>yarn.nodemanager.remote-app-log-dir</name>
		<value>/home/hadoop/logs</value>
	</property>
	
	<property>
		<description>Number of CPU cores that can be allocated for containers.</description>
		<name>yarn.nodemanager.resource.cpu-vcores</name>
		<value>4</value>
	</property>
	<property>
		<description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>

5.1.5修改slaves 

bigdatasvr01
bigdatasvr03

5.2 启动journalnode(每个节点上都运行)

5.3 格式化namenode(nn1)

5.4 启动namenode (nn1)

5.5格式化namenode(nn2)