CDH安装: tar包+parcel a, 启动server服务b, 启动agent 服务3, 安装hive, sqoop4, 安装kafka5，安装huecloudera-server 安装过程

96 阅读 0 评论 64 点赞

我是靠谱客的博主爱撒娇树叶，这篇文章主要介绍CDH安装: tar包+parcel a, 启动server服务b, 启动agent 服务3, 安装hive, sqoop4, 安装kafka5，安装huecloudera-server 安装过程，现在分享给大家，希望可以做个参考。

安装指南：https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/installation.html

问题1：解决root用户在hdfs中permission denied问题

- 方法1：修改hdfs配置中的supergroup超级组 (默认值为supergroup, 改为root,重启配置就行)

- 方法2：修改hdfs配置中的System User： root (不推荐）

cloudera安装hdfs后，默认的配置System User是hdfs, 修改root用户，重启会报错：/var/run/hdfs-sockets权限问题（由于是旧的用户占用了socket,所以删除就行）

复制代码

1
2
3
4
5
6
#java.io.IOException: the path component: '/var/run/hdfs-sockets' is owned by a user who is not root and not you.
#Your effective user id is 0; the path is owned by user id 496, and its permissions are 0755.  Please fix this or select a different socket path

#解决：
rm -rf  /var/run/hdfs-sockets/

问题2：修改hdfs系统用户System User为普通用户: wang

复制代码

1
2
3
chown -R wang /dfs
chown -R wang /var/log/hadoop-hdfs
chown -R wang /var/run/hdfs-sockets

问题3：修改主机名：确认主机名没有 _ (下划线)

复制代码

1
2
 主机名不合法，存在_时，agent启动后一会又stop了，日志打印为：
Re-using pre-existing directory: /var/run/cloudera-scm-agent

脚本化快速安装cdh集群，参考另一篇博客：https://blog.csdn.net/eyeofeagle/article/details/98667232

要点：

工具的特点
具体操作

1，什么是CM? (Cloudera Manager)

下载地址： http://archive.cloudera.com/cm5/cm/5/

Cloudera Manager则是为了便于在集群中进行Hadoop等大数据处理相关的服务安装和监控管理的组件，对集群中主机、Hadoop、Hive、Spark等服务的安装配置管理做了极大简化。

CDH (Cloudera's Distribution, including Apache Hadoop)，是Hadoop众多分支中的一种，由Cloudera维护，基于稳定版本的Apache Hadoop构建，并集成了很多补丁，可直接用于生产环境。

2，如何使用CM来安装CDH ?

下载地址： http://archive.cloudera.com/cdh5/parcels

(安装包版本需要和系统的版本对应，此处使用集群： centos7 × 3 【一个namenode, 两个datenode】 )

准备CM安装包: cloudera-manager-centos7-cm5.13.0_x86_64.tar.gz,
CDH安装包: CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel， CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1

cloudera-manager的tar包，解压目录为：（红色部分为不存在的，需要手动创建）

3，具体安装步骤：先安装CM, 启动server + agent--->进入网页：namenode:7180

安装mysql: [上一篇博客： https://blog.csdn.net/eyeofeagle/article/details/85010957]
配置cdh : root用户登陆在/root目录下准备文件【cm, cdh】
a, 启动server服务

复制代码

###########################################
#root 家目录准备的文件: 
#[root@s101 ~]# ls
#CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
#CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha
#manifest.json
#cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

#jdk-7u80-linux-x64.tar.gz
#mysql-connector-java-5.1.27.jar
###########################################

#关闭防火墙
systemctl stop firewalld
service firewalld stop
service firewalld status
 
#删除原配置
for u in cloudera-scm flume hadoop hdfs hbase hive httpfs hue impala llama mapred oozie solr spark sqoop sqoop2 yarn zookeeper; do sudo kill $(ps -u $u -o pid=); done
sudo umount cm_processes
sudo rm -Rf /opt/cm* /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*

sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
sudo rm -Rf /dfs  /yarn
mysql -uroot -p1234 -e "drop database scm; drop database hue; drop database hive; drop database oozie; "

# 修改提供服务的server ip
tar -zxvf ~/cloudera-manager*.tar.gz  -C /opt/
cp ~/CDH*.parcel* /opt/cloudera/parcel-repo/
cp ~/manifest.json /opt/cloudera/parcel-repo/
#sed -i s@server_host=localhost@server_host=s101@g    /opt/cm*/etc/cloudera-scm-agent/config.ini
 
 
#1,  准备jdk
existsJdk=`whereis java |awk -F"java:" '{print $2}' `
[ ! $existsJdk ] &&  mkdir /usr/java ;
tar -zxvf *jdk*.gz -C /usr/java/ && cd /usr/java/; ln -sfT *jdk* jdk; 
cat >> /etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
EOF
source /etc/profile

2,  创建cloudera-scm用户: (在所有节点, 包括agent, server )
userdel cloudera-scm
cd /opt/cm*
useradd --system --home=`pwd`/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
 
#3, 配置mysql 元数据信息： cloudera-manager使用mysql保存hive, oozie等的元数据
#修改mysql密码： mysqladmin -u root password '123456'
[ ! -d /usr/share/java ] &&  mkdir /usr/share/java -p
cp ~/mysql-connector-*.jar /usr/share/java/mysql-connector-java.jar
cp ~/mysql-connector-*.jar  /opt/cm*/share/cmf/lib
mysql -uroot -p123456 -e "drop database if exists hive ; create database hive default charset utf8";
mysql -uroot -p123456 -e "drop database if exists zookeeper ; create database zookeeper default charset utf8";
mysql -uroot -p123456 -e "drop database if exists oozie; create database oozie default charset utf8";
mysql -uroot -p123456 -e "drop database if exists hue; create database hue default charset utf8";
mysql -uroot -p123456 -e "drop database if exists scm";

#mysql -uroot -p12345 --scm-host [cm server的主机名] [cm的数据库]   [cm数据库访问用户] [cm数据库访问用户的密码]
/opt/cm*/share/cmf/schema/scm_prepare_database.sh mysql  -uroot -p123456  scm scm scm
#或者直接编辑/opt/cm*/etc/cloudera-scm-server/db.properties, 配置mysql连接的参数
#如果要修改mysql端口：　com.cloudera.cmf.db.host=localhost:3308

mysql -uroot -p123456 -e "grant all privileges on *.* to 'scm'@'%' identified by 'scm' with grant option; flush privileges;";
mysql -uroot -p123456 -e "grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option; flush privileges;";
 
 
#4, 更改/opt目录为此用户所属
chown -R cloudera-scm:cloudera-scm /opt/{cloudera,cm*}
[ ! -d /var/lib/cloudera-scm-server ] && mkdir /var/lib/cloudera-scm-server
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
 
#5，启动server
/opt/cm*/etc/init.d/cloudera-scm-server start
tail -F /opt/cm*/log/cloudera-scm-server/cloudera-scm-server.log 
 
 #启动agent
[ ! -d /opt/cloudera/parcels ]  && mkdir  /opt/cloudera/parcels -p 
[ ! -d /opt/cloudera/csd ]  	&& mkdir  /opt/cloudera/csd -p 
chown -R cloudera-scm:cloudera-scm /opt/
/opt/cm*/etc/init.d/cloudera-scm-agent start
tail -F /opt/cm*/log/cloudera-scm-agent/cloudera-scm-agent.log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
###########################################
#root 家目录准备的文件: 
#[root@s101 ~]# ls
#CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
#CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha
#manifest.json
#cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

#jdk-7u80-linux-x64.tar.gz
#mysql-connector-java-5.1.27.jar
###########################################

#关闭防火墙
systemctl stop firewalld
service firewalld stop
service firewalld status
 
#删除原配置
for u in cloudera-scm flume hadoop hdfs hbase hive httpfs hue impala llama mapred oozie solr spark sqoop sqoop2 yarn zookeeper; do sudo kill $(ps -u $u -o pid=); done
sudo umount cm_processes
sudo rm -Rf /opt/cm* /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*

sudo rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
sudo rm -Rf /dfs  /yarn
mysql -uroot -p1234 -e "drop database scm; drop database hue; drop database hive; drop database oozie; "

# 修改提供服务的server ip
tar -zxvf ~/cloudera-manager*.tar.gz  -C /opt/
cp ~/CDH*.parcel* /opt/cloudera/parcel-repo/
cp ~/manifest.json /opt/cloudera/parcel-repo/
#sed -i s@server_host=localhost@server_host=s101@g    /opt/cm*/etc/cloudera-scm-agent/config.ini
 
 
#1,  准备jdk
existsJdk=`whereis java |awk -F"java:" '{print $2}' `
[ ! $existsJdk ] &&  mkdir /usr/java ;
tar -zxvf *jdk*.gz -C /usr/java/ && cd /usr/java/; ln -sfT *jdk* jdk; 
cat >> /etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
EOF
source /etc/profile


2,  创建cloudera-scm用户: (在所有节点, 包括agent, server )
userdel cloudera-scm
cd /opt/cm*
useradd --system --home=`pwd`/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
 
#3, 配置mysql 元数据信息： cloudera-manager使用mysql保存hive, oozie等的元数据
#修改mysql密码： mysqladmin -u root password '123456'
[ ! -d /usr/share/java ] &&  mkdir /usr/share/java -p
cp ~/mysql-connector-*.jar /usr/share/java/mysql-connector-java.jar
cp ~/mysql-connector-*.jar  /opt/cm*/share/cmf/lib
mysql -uroot -p123456 -e "drop database if exists hive ; create database hive default charset utf8";
mysql -uroot -p123456 -e "drop database if exists zookeeper ; create database zookeeper default charset utf8";
mysql -uroot -p123456 -e "drop database if exists oozie; create database oozie default charset utf8";
mysql -uroot -p123456 -e "drop database if exists hue; create database hue default charset utf8";
mysql -uroot -p123456 -e "drop database if exists scm";

#mysql -uroot -p12345 --scm-host [cm server的主机名] [cm的数据库]   [cm数据库访问用户] [cm数据库访问用户的密码]
/opt/cm*/share/cmf/schema/scm_prepare_database.sh mysql  -uroot -p123456  scm scm scm
#或者直接编辑/opt/cm*/etc/cloudera-scm-server/db.properties, 配置mysql连接的参数
#如果要修改mysql端口：　com.cloudera.cmf.db.host=localhost:3308

mysql -uroot -p123456 -e "grant all privileges on *.* to 'scm'@'%' identified by 'scm' with grant option; flush privileges;";
mysql -uroot -p123456 -e "grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option; flush privileges;";
 
 
#4, 更改/opt目录为此用户所属
chown -R cloudera-scm:cloudera-scm /opt/{cloudera,cm*}
[ ! -d /var/lib/cloudera-scm-server ] && mkdir /var/lib/cloudera-scm-server
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
 
#5，启动server
/opt/cm*/etc/init.d/cloudera-scm-server start
tail -F /opt/cm*/log/cloudera-scm-server/cloudera-scm-server.log 
 
 #启动agent
[ ! -d /opt/cloudera/parcels ]  && mkdir  /opt/cloudera/parcels -p 
[ ! -d /opt/cloudera/csd ]  	&& mkdir  /opt/cloudera/csd -p 
chown -R cloudera-scm:cloudera-scm /opt/
/opt/cm*/etc/init.d/cloudera-scm-agent start
tail -F /opt/cm*/log/cloudera-scm-agent/cloudera-scm-agent.log

b, 启动agent 服务

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
tar -zxvf ~/cloudera-manager*.tar.gz -C /opt
sed -i s@server_host=localhost@server_host=s222@g    /opt/cm*/etc/cloudera-scm-agent/config.ini
mkdir  /opt/cloudera/parcels -p 
 
#创建用户
userdel cloudera-scm
cd /opt/cm*
useradd --system --home=`pwd`/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
chown -R cloudera-scm:cloudera-scm /opt/
 
# 启动agent
/opt/cm-*/etc/init.d/cloudera-scm-agent start
tail -F /opt/cm*/log/cloudera-scm-agent/cloudera-scm-agent.log

3, 安装hive, sqoop

复制代码

1
2
3
4
5
6
7
#安装hive的机器
sudo cp mysql-connector-java-5.1.27.jar /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hive/lib/
sudo cp mysql-connector-java-5.1.27.jar  /usr/share/java/mysql-connector-java.jar

#安装sqoop的机器
sudo cp mysql-connector-java-5.1.27.jar /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/sqoop/lib/
sudo cp mysql-connector-java-5.1.27.jar  /usr/share/java/mysql-connector-java.jar

4, 安装kafka

复制代码

1
2
3
4
5
6
7
8
9
下载kafka的parcel文件： http://archive.cloudera.com/kafka/parcels/latest/，放在/opt/cloudera/parcel-repo下
下载kafka jar包,下载地址：http://archive.cloudera.com/csds/kafka/, 放在/opt/cloudera/csd下

kafka启动报错： 在报错中有outofMemory的提示==> 解决： 在cdh:7180界面搜索kafka配置的memory选项，修改 Java Heap Size of Broker为1G

kafka生产者发送消息报错： kafka-console-producer --broker-list localhost:9092 --topic t1
 ERROR Error when sending message to topic t1 with key: null, value: 2 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
=======》解决： 输入的zookeeper或broker的地址localhost解析为127.0.0.1找不到对应的zk服务，所以修改为正确的主机名

5，安装hue

https://blog.csdn.net/eyeofeagle/article/details/86522497

cloudera-server 安装过程

1 ，cm-server 安装时消除警告

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#=====临时修改swappiness
sysctl -w vm.swappiness=10
#====永久修改swappiness
# echo "vm.swappiness=10" >> /etc/sysctl.conf

#刷新交换空间配置
swapoff -a
swapon -a


#页面压缩
chmod +x  /etc/rc.d/rc.local
echo  'echo never > /sys/kernel/mm/transparent_hugepage/defrag ' >>/etc/rc.d/rc.local
echo  'echo never > /sys/kernel/mm/transparent_hugepage/enabled '>>/etc/rc.d/rc.local

2，server 安装日志

复制代码

[11/Dec/2018 08:09:14 +0000] 10990 MainThread __init__     INFO     Importing metric schema from file /opt/cm-5.10.0/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/schema.json

[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        WARNING  Expected user root for /opt/cloudera/parcels but was cloudera-scm
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        WARNING  Expected group root for /opt/cloudera/parcels but was cloudera-scm

#准备torrent文件
2018-12-14 00:22:10,503 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2018-12-14 00:22:11,333 INFO WebServerImpl:org.springframework.web.servlet.DispatcherServlet: FrameworkServlet 'Spring MVC Dispatcher Servlet': initialization completed in 31605 ms
2018-12-14 00:22:11,576 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2018-12-14 00:22:14,410 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2018-12-14 00:22:18,868 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (1482270967 bytes) in 2828 pieces (2828 expected) in 41626.3ms.

1
2
3
4
5
6
7
8
9
10
11
12
[11/Dec/2018 08:09:14 +0000] 10990 MainThread __init__     INFO     Importing metric schema from file /opt/cm-5.10.0/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/schema.json

[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        WARNING  Expected user root for /opt/cloudera/parcels but was cloudera-scm
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        WARNING  Expected group root for /opt/cloudera/parcels but was cloudera-scm

#准备torrent文件
2018-12-14 00:22:10,503 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2018-12-14 00:22:11,333 INFO WebServerImpl:org.springframework.web.servlet.DispatcherServlet: FrameworkServlet 'Spring MVC Dispatcher Servlet': initialization completed in 31605 ms
2018-12-14 00:22:11,576 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2018-12-14 00:22:14,410 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2018-12-14 00:22:18,868 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (1482270967 bytes) in 2828 pieces (2828 expected) in 41626.3ms.

3，agent 安装日志

复制代码

[11/Dec/2018 08:09:14 +0000] 10990 MainThread agent        ERROR    Failed to connect to previous supervisor.
===解决： ps -Af |grep supervisor |awk '{print $2 }' |xargs -n 1 kill -9 
重启agent即可
 
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        INFO     CM server guid: 66c038f8-ec74-4264-9512-c43da970dd2f
[12/Dec/2018 15:28:55 +0000] 11816 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[12/Dec/2018 15:28:55 +0000] 11816 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[12/Dec/2018 15:28:56 +0000] 11816 MainThread agent        ERROR    Error, CM server guid updated, expected 80184fab-b100-471c-a031-419969f7d72b, received 77ffb456-1f52-4a58-a940-f4a67ee7053c
===删除文件： /opt/cm-5.*/lib/cloudera-scm-agent/cm_guid

1,[11/Dec/2018 08:09:18 +0000] 10990 MainThread downloader   ERROR    Failed rack peer update: [Errno 111] Connection refused
2，[11/Dec/2018 08:11:36 +0000] 10990 Thread-13 downloader   ERROR    Failure due to stall on seeded torrent.

2,[root@s103 wang]# cat /opt/cm*/log/cloudera-scm-agent/cloudera-scm-agent.out
[12/Dec/2018 08:22:24 +0000] 4445 MainThread agent        INFO     SCM Agent Version: 5.8.0
Unable to create the pidfile.
====解决： mkdir /opt/cm-5.7.0/run/cloudera-scm-agent; chown -R cloudera-scm:cloudera-scm /opt/;

3， #下载parcel文件到  /opt/cloudera/parcels/.flood
[12/Dec/2018 08:45:58 +0000] 5474 CP Server Thread-4 _cplogging   INFO     192.168.56.113 - - [12/Dec/2018:08:45:58] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[12/Dec/2018 08:46:02 +0000] 5474 Thread-13 downloader   INFO     Fetching torrent: http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel.torrent
[12/Dec/2018 08:46:02 +0000] 5474 Thread-13 downloader   INFO     Starting download of: http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel
[12/Dec/2018 08:46:03 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=0 totalSize=1486884451 upload=0 state=checking_resume_data seed=[] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=0]
=======parcels下载进度： 一直为0 ==>解决： 删除/opt/cloudera/{csd,parcel-cache,parcels/.flood/}/*; 重启服务

[12/Dec/2018 08:46:03 +0000] 5474 Thread-13 downloader   INFO     Setting url seed: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel ([])
[12/Dec/2018 08:46:05 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=11534336 totalSize=1486884451 upload=0 state=downloading seed=['http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel'] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=7757]
[12/Dec/2018 08:46:07 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=26214400 totalSize=1486884451 upload=0 state=downloading seed=['http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel'] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=17630]

4， #解压parcel文件 ： 到/opt/cloudera/parcels
12/Dec/2018 08:48:38 +0000] 5474 Thread-13 downloader   INFO     Completed download of http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel code=200 state=downloaded
[12/Dec/2018 08:48:38 +0000] 5474 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[12/Dec/2018 08:48:38 +0000] 5474 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel into /opt/cloudera/parcels

5， #添加用户：
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.8.0-1.cdh5.8.0.p0.42
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Ensuring users/groups exist for new parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'hadoop']
[12/Dec/2018 08:50:57 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'flume']
[12/Dec/2018 08:50:57 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'hdfs']

[12/Dec/2018 08:50:58 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'flume', '-K', 'UMASK=022', '--home', '/var/lib/flume-ng', '--comment', 'Flume', '--shell', '/bin/false', 'flume']
[12/Dec/2018 08:50:58 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hdfs', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-hdfs', '--comment', 'Hadoop HDFS', '--shell', '/bin/bash', '--groups', 'hadoop', 'hdfs']

6， #修改文件：用户所属权限
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Ensuring correct file permissions for new parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     chown: /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml oozie oozie
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml']
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     chmod: /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml 0750
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml']

7， #执行alternatives.sh
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Ensuring alternatives entries are activated for parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Executing command ['/opt/cm-5.8.0/lib64/cmf/service/common/alternatives.sh', 'activate', 'sqoop-import', '/usr/bin/sqoop-import', 'bin/sqoop-import', '10', 'False']
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Executing command ['/opt/cm-5.8.0/lib64/cmf/service/common/alternatives.sh', 'activate', 'solr-conf', '/etc/solr/conf', 'etc/solr/conf.dist', '10', 'True']

8， #加载manifest.json文件, 完成安装
[12/Dec/2018 08:51:17 +0000] 5474 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[12/Dec/2018 08:51:17 +0000] 5474 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.8.0-1.cdh5.8.0.p0.42
[12/Dec/2018 08:55:47 +0000] 5474 CP Server Thread-7 _cplogging   INFO     192.168.56.113 - - [12/Dec/2018:08:55:47] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[12/Dec/2018 08:55:48 +0000] 5474 MainThread heartbeat_tracker INFO     HB stats (seconds): num:44 LIFE_MIN:0.14 min:0.02 mean:0.25 max:3.35 LIFE_MAX:0.40

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
[11/Dec/2018 08:09:14 +0000] 10990 MainThread agent        ERROR    Failed to connect to previous supervisor.
===解决： ps -Af |grep supervisor |awk '{print $2 }' |xargs -n 1 kill -9 
重启agent即可
 
[11/Dec/2018 08:09:18 +0000] 10990 MainThread agent        INFO     CM server guid: 66c038f8-ec74-4264-9512-c43da970dd2f
[12/Dec/2018 15:28:55 +0000] 11816 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[12/Dec/2018 15:28:55 +0000] 11816 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[12/Dec/2018 15:28:56 +0000] 11816 MainThread agent        ERROR    Error, CM server guid updated, expected 80184fab-b100-471c-a031-419969f7d72b, received 77ffb456-1f52-4a58-a940-f4a67ee7053c
===删除文件： /opt/cm-5.*/lib/cloudera-scm-agent/cm_guid

1,[11/Dec/2018 08:09:18 +0000] 10990 MainThread downloader   ERROR    Failed rack peer update: [Errno 111] Connection refused
2，[11/Dec/2018 08:11:36 +0000] 10990 Thread-13 downloader   ERROR    Failure due to stall on seeded torrent.

2,[root@s103 wang]# cat /opt/cm*/log/cloudera-scm-agent/cloudera-scm-agent.out
[12/Dec/2018 08:22:24 +0000] 4445 MainThread agent        INFO     SCM Agent Version: 5.8.0
Unable to create the pidfile.
====解决： mkdir /opt/cm-5.7.0/run/cloudera-scm-agent; chown -R cloudera-scm:cloudera-scm /opt/; 

3， #下载parcel文件到  /opt/cloudera/parcels/.flood
[12/Dec/2018 08:45:58 +0000] 5474 CP Server Thread-4 _cplogging   INFO     192.168.56.113 - - [12/Dec/2018:08:45:58] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[12/Dec/2018 08:46:02 +0000] 5474 Thread-13 downloader   INFO     Fetching torrent: http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel.torrent
[12/Dec/2018 08:46:02 +0000] 5474 Thread-13 downloader   INFO     Starting download of: http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel
[12/Dec/2018 08:46:03 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=0 totalSize=1486884451 upload=0 state=checking_resume_data seed=[] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=0]
=======parcels下载进度： 一直为0 ==>解决： 删除/opt/cloudera/{csd,parcel-cache,parcels/.flood/}/*; 重启服务


[12/Dec/2018 08:46:03 +0000] 5474 Thread-13 downloader   INFO     Setting url seed: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel ([])
[12/Dec/2018 08:46:05 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=11534336 totalSize=1486884451 upload=0 state=downloading seed=['http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel'] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=7757]
[12/Dec/2018 08:46:07 +0000] 5474 Thread-13 downloader   INFO     Current state: CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel [totalDownloaded=26214400 totalSize=1486884451 upload=0 state=downloading seed=['http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel'] location=/opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel progress=17630]

4， #解压parcel文件 ： 到/opt/cloudera/parcels
12/Dec/2018 08:48:38 +0000] 5474 Thread-13 downloader   INFO     Completed download of http://s103:7180/cmf/parcel/download/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel code=200 state=downloaded
[12/Dec/2018 08:48:38 +0000] 5474 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[12/Dec/2018 08:48:38 +0000] 5474 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel/CDH-5.8.0-1.cdh5.8.0.p0.42-el7.parcel into /opt/cloudera/parcels

5， #添加用户：
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.8.0-1.cdh5.8.0.p0.42
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Ensuring users/groups exist for new parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:50:56 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'hadoop']
[12/Dec/2018 08:50:57 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'flume']
[12/Dec/2018 08:50:57 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'hdfs']

[12/Dec/2018 08:50:58 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'flume', '-K', 'UMASK=022', '--home', '/var/lib/flume-ng', '--comment', 'Flume', '--shell', '/bin/false', 'flume']
[12/Dec/2018 08:50:58 +0000] 5474 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'hdfs', '-K', 'UMASK=022', '--home', '/var/lib/hadoop-hdfs', '--comment', 'Hadoop HDFS', '--shell', '/bin/bash', '--groups', 'hadoop', 'hdfs']

6， #修改文件：用户所属权限
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Ensuring correct file permissions for new parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     chown: /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml oozie oozie
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml']
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     chmod: /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml 0750
[12/Dec/2018 08:51:00 +0000] 5474 MainThread parcel       INFO     Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/etc/oozie/tomcat-conf.https/conf/server.xml']

7， #执行alternatives.sh
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Ensuring alternatives entries are activated for parcel CDH-5.8.0-1.cdh5.8.0.p0.42.
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Executing command ['/opt/cm-5.8.0/lib64/cmf/service/common/alternatives.sh', 'activate', 'sqoop-import', '/usr/bin/sqoop-import', 'bin/sqoop-import', '10', 'False']
[12/Dec/2018 08:51:14 +0000] 5474 MainThread parcel       INFO     Executing command ['/opt/cm-5.8.0/lib64/cmf/service/common/alternatives.sh', 'activate', 'solr-conf', '/etc/solr/conf', 'etc/solr/conf.dist', '10', 'True']

8， #加载manifest.json文件, 完成安装
[12/Dec/2018 08:51:17 +0000] 5474 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[12/Dec/2018 08:51:17 +0000] 5474 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.8.0-1.cdh5.8.0.p0.42
[12/Dec/2018 08:55:47 +0000] 5474 CP Server Thread-7 _cplogging   INFO     192.168.56.113 - - [12/Dec/2018:08:55:47] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[12/Dec/2018 08:55:48 +0000] 5474 MainThread heartbeat_tracker INFO     HB stats (seconds): num:44 LIFE_MIN:0.14 min:0.02 mean:0.25 max:3.35 LIFE_MAX:0.40

大数据parel组件：安装异常处理

1 ，安装hdfs， datanode启动不了

查看datanode启动日志：默认路径/var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-s101.log.out

复制代码

java.io.IOException: the path component: '/var/run/hdfs-sockets' is owned by a user who is not root and not you.  Your effective user id is 978; the path is owned by user id 996, and its permissions are 0755.  Please fix this or select a different socket path.
        at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
        at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
        at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1072)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1038)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1264)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:464)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2545)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2432)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2479)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2661)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2685)
2018-12-11 05:18:44,948 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-12-11 05:18:44,952 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

Caused by: java.lang.ExceptionInInitializerError
	at com.cloudera.cmf.service.config.CMURLEvaluator.<clinit>(CMURLEvaluator.java:44)
	at com.cloudera.cmf.service.mgmt.NavMetaServerParams.<clinit>(NavMetaServerParams.java:833)
	at com.cloudera.server.web.cmf.DbTestConnUtil.<clinit>(DbTestConnUtil.java:92)
	at com.cloudera.server.cmf.components.EmbeddedDbManager.<init>(EmbeddedDbManager.java:137)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
	... 63 more
Caused by: java.net.UnknownHostException: bogon: bogon: 未知的名称或服务
	at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
	at com.cloudera.cmf.service.config.CMURLEvaluator.<clinit>(CMURLEvaluator.java:42)
	... 71 more
Caused by: java.net.UnknownHostException: bogon: 未知的名称或服务
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
	... 72 more
===解决：( 由centos7 的cdh镜像，复制到新的centos7机器运行cdh) 
127.0.0.1   localhost localhost.localdomain  bongon s222

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
java.io.IOException: the path component: '/var/run/hdfs-sockets' is owned by a user who is not root and not you.  Your effective user id is 978; the path is owned by user id 996, and its permissions are 0755.  Please fix this or select a different socket path.
        at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
        at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
        at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1072)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1038)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1264)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:464)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2545)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2432)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2479)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2661)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2685)
2018-12-11 05:18:44,948 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-12-11 05:18:44,952 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

Caused by: java.lang.ExceptionInInitializerError
	at com.cloudera.cmf.service.config.CMURLEvaluator.<clinit>(CMURLEvaluator.java:44)
	at com.cloudera.cmf.service.mgmt.NavMetaServerParams.<clinit>(NavMetaServerParams.java:833)
	at com.cloudera.server.web.cmf.DbTestConnUtil.<clinit>(DbTestConnUtil.java:92)
	at com.cloudera.server.cmf.components.EmbeddedDbManager.<init>(EmbeddedDbManager.java:137)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
	... 63 more
Caused by: java.net.UnknownHostException: bogon: bogon: 未知的名称或服务
	at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
	at com.cloudera.cmf.service.config.CMURLEvaluator.<clinit>(CMURLEvaluator.java:42)
	... 71 more
Caused by: java.net.UnknownHostException: bogon: 未知的名称或服务
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
	... 72 more
===解决：( 由centos7 的cdh镜像，复制到新的centos7机器运行cdh) 
127.0.0.1   localhost localhost.localdomain  bongon s222