【CDH部署】全网最简单CDH6.2.0环境部署

下载链接

官方下载链接: https://archive.cloudera.com/cdh6/6.2.0/parcels/
百度网盘下载链接: https://pan.baidu.com/s/1Dm5Elf9uQqn14BUbgU3AFQ 提取码: mws3

主机规划
sparkproject1 manager 8G 50GB
sparkproject2 agent 4G 50GB
sparkproject3 agent 4G 50GB

1. 设置IP, hostname, hosts, ssh免密登录

可参考Hadoop中linux基本配置那篇文章
【Hadoop部署】Hadoop环境部署1-Linux基础设置

2. 关闭防火墙

systemctl stop firewalld 关闭防火墙
systemctl disable firewalld 禁止防火墙开机自启
vim /etc/selinux/config —> SELINUX=disabled

3.安装java, 和mysql-connector-java-5.1.34.jar

========

4.一定要装mysql, 不能用mariadb会有依赖包的问题

删除mysql旧的依赖
rpm -qa|grep mariadb
yum remove mariadb

安装mysql

rpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm

初始化数据库，会生成随机密码
mysqld --initialize --user=root
cat /var/log/mysqld.log | tail -n5
末尾有密码：abCIja+g1aJ?

修改mysql配置文件
vim /etc/my.cnf
在 [mysqld] 标签下添加

init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake

chmod -R 777 /var/lib/mysql
systemctl start mysqld.service
systemctl enable mysqld.service

进入CLI环境，修改root默认密码。创建必要的表，更新远程访问权限
mysql -uroot -p

修改用户密码

ALTER USER USER() IDENTIFIED BY 'xiangyang';

创建后面cdh需要的表

create database cmserver default charset utf8 collate utf8_general_ci;
create database metadata default charset utf8 collate utf8_general_ci;
create database oozie default charset utf8 collate utf8_general_ci;
create database hue default charset utf8 collate utf8_general_ci;

允许远程访问mysql

update user set host='%' where host='localhost';
flush privileges;

systemctl restart mysqld

以下是模拟Cloudera的仓库环境，本地安装cdh
安装apache服务器
yum install httpd
systemctl start httpd.service
systemctl enable httpd.service

yum install -y createrepo
mkdir -p /var/www/html/cloudera-repos/cm6
cd /var/www/html/cloudera-repos/cm6/
cm文件考过来
createrepo /var/www/html/cloudera-repos/cm6/

yum clean all
yum makecache

vim /etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]
name = Cloudera Manager, Version 6.2.0
baseurl = http://sparkproject1/cloudera-repos/cm6/
gpgcheck = 0

yum clean all
yum makecache

rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera
sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

cd /opt/cloudera/parcel-repo
parcel拷过来

注册数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmserver root xiangyang

启动Cloudera Manager Server

大概要几分钟。可以观察/var/log/cloudera_server中的日志，看有没有报错
service cloudera-scm-server start

成功的话关机克隆，更新一些必要信息
ifconfig ens33 192.168.71.111
vim /etc/sysconfig/network-scripts/ifcfg-ens33
vim /etc/hostname
hostnamectl set-hostname sparkproject2

子节点可以删的组件

yum remove mysql
yum remove httpd
yum remove cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
rm -rf /var/www/html/cloudera-repos/cm6/
rm -rf /var/www/html/streamsets

问题答疑

如果发现没有权限操作hdfs的情况，使用如下命令

sudo -u hdfs hadoop fs -setrep -R 2 /

StreamSets的安装

拷贝STREAMSETS-3.13.0.jar 到 /opt/cloudera/csd
mkdir -p /var/www/html/streamsets
streamsets文件拷过来
createrepo /var/www/html/streamsets

vim /etc/yum.repos.d/ss.repo

[ssrepo]
name = ss_repo
baseurl = http://sparkproject1/streamsets
enable = true
gpgcheck = false

扩展磁盘

pvcreate /dev/mapper/centos-root
lvextend -L +10G /dev/mapper/centos-root

sync; echo 3 > /proc/sys/vm/drop_caches ; free -h

最后

以上就是合适含羞草为你收集整理的【CDH部署】全网最简单CDH6.2.0环境部署的全部内容，希望文章能够帮你解决【CDH部署】全网最简单CDH6.2.0环境部署所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。