cdh集群安装sqoop

78 阅读 0 评论 52 点赞

我是靠谱客的博主魁梧春天，最近开发中收集的这篇文章主要介绍cdh集群安装sqoop，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

sqoop-cdh安装：

1、上传sqoop-1.4.6-cdh5.10.1.tar.gz包到/opt/sqoop目录
2、解压缩文件： tar -zxf sqoop-1.4.6-cdh5.10.1.tar.gz
3、设置环境变量：
vim /etc/profile
#添加一下内容
export JAVA_HOME=/usr/java/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH

#Kylin
#export KYLIN_HOME=/opt/kylin/apache-kylin-2.1.0-bin-cdh57/

#Hadoop
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop
export HADOOP_CMD=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop
export PATH=$PATH:$HADOOP_HOME:$HADOOP_HOME/bin

#hbasae
export HBASE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hbase
export PATH=$PATH:$HBASE_HOME:$HBASE_HOME/bin

#hive
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hive
export HCAT_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hive-hcatalog
export PATH=$PATH:$HIVE_HOME:$HIVE_HOME/bin

#sqoop
export SQOOP_HOME=/opt/sqoop/sqoop-1.4.6-cdh5.10.1
export PATH=$PATH:$SQOOP_HOME/bin

4、修改sqoop配置文件：
进入到conf目录：cd /opt/sqoop/sqoop-1.4.6-cdh5.10.1/conf
修改文件名：cp sqoop-env-template.sh sqoop-env.sh
修改文件内容：vim sqoop-env.sh
#修改内容如下：
# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop

#set the path to where bin/hbase is available
export HBASE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hive

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/zookeeper

5、导入驱动包

sqoop安装完成后,根据连接方式不同，选择导入jar包：

1、mysql连接导入jar:

将jar包上传到/opt/sqoop目录
执行命令：
拷贝到$SQOOP_HOME/lib目录 cp /opt/sqoop/mysql-connector-java-5.1.17.jar /opt/sqoop/sqoop-1.4.6-cdh5.10.1/lib
拷贝到$HADOOP_HOME/lib目录 cp /opt/sqoop/mysql-connector-java-5.1.17.jar /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/lib

2、sqlserver连接导入jar:
执行命令：
拷贝到$SQOOP_HOME/lib目录 cp /opt/sqoop/sqljdbc4-4.0.jar /opt/sqoop/sqoop-1.4.6-cdh5.10.1/lib
拷贝到$HADOOP_HOME/lib目录 cp /opt/sqoop/sqljdbc4-4.0.jar /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/lib

注意：驱动包在导入到sqoop_home/lib目录后，通过sqoop连接数据库异常找不到驱动包，所以要在此将驱动包导入到HADOOP_HOME/lib目录下。

6、验证结果
1、mysql执行：
sqoop list-databases --connect jdbc:mysql://192.168.3.76:3306/ --username root -password 123456
执行结果：
[root@node5 sqoop]# sqoop list-databases --connect jdbc:mysql://192.168.3.76:3306/ --username root -password 123456
Warning: /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/01/06 15:38:35 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0
18/01/06 15:38:35 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/01/06 15:38:35 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
test
[root@node5 sqoop]#

2、sqlserver执行:
sqoop list-databases --connect 'jdbc:sqlserver://192.168.2.219; username=nsmc53; password=123456;database=WFBDCMain'
执行结果：
[root@node5 sqoop]# sqoop list-databases --connect 'jdbc:sqlserver://192.168.2.219; username=nsmc53; password=123456;database=WFBDCMain'
Warning: /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/01/06 15:39:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0
18/01/06 15:39:16 INFO manager.SqlManager: Using default fetchSize of 1000
master
tempdb
model
MaintenanceLog2017
MaintenanceLog2018
WFBDCMainLog2017
WFBDCMainLog2018
WDBDCStatLog201711
WDBDCStatLog201712
WDBDCStatLog201801
NSMCStatLog201711
NSMCStatLog201712
NSMCStatLog201801
ISIPStatLog2017
ISIPStatLog2018
WDBDCStatLog2017
WDBDCStatLog2018
[root@node5 sqoop]#