Hadoop组件sqoop部署

75 阅读 0 评论 50 点赞

我是靠谱客的博主魁梧衬衫，这篇文章主要介绍Hadoop组件sqoop部署，现在分享给大家，希望可以做个参考。

1.安装sqoop

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#解压并改名
[root@master app]# tar -zxvf sqoop-1.4.6-cdh5.5.2.tar -C /app/
[root@master app]# mv sqoop-1.4.6-cdh5.5.2/ sqoop
#配置环境变量
#注意自己的目录
[root@master sqoop]# vim /etc/profile
export SQOOP_HOME=/app/sqooq
export PATH=$PATH:$SQOOP_HOME/bin
export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib
#将mysql的jar包放入sqoop的lib目录中
[root@master home]# cp mysql-connector-java-5.1.39.jar /app/sqoop/lib
#编辑配置文件
#文件内容
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/app/hadoop
#Hadoop安装目录
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/app/hadoop
#Hadoop安装目录
#set the path to where bin/hbase is available
export HBASE_HOME=/app/hbase
#hbase安装目录
#Set the path to where bin/hive is available
export HIVE_HOME=/app/hive
#hive安装目录
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=
#可不配置

2.验证是否成功

复制代码

[root@master conf]# sqoop version
Warning: /app/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /app/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/03/12 20:29:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.2
Sqoop 1.4.6-cdh5.5.2
git commit id 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Mon Jan 25 16:10:15 PST 2016
#连接数据库测试
[root@master conf]# sqoop-list-databases --connect jdbc:mysql://master:3306/ -username root -password 123456
Warning: /app/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /app/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/03/12 20:31:03 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.2
21/03/12 20:31:03 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/03/12 20:31:04 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Fri Mar 12 20:31:04 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sys

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@master conf]# sqoop version
Warning: /app/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /app/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/03/12 20:29:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.2
Sqoop 1.4.6-cdh5.5.2
git commit id 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Mon Jan 25 16:10:15 PST 2016
#连接数据库测试
[root@master conf]# sqoop-list-databases --connect jdbc:mysql://master:3306/ -username root -password 123456
Warning: /app/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /app/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/03/12 20:31:03 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.2
21/03/12 20:31:03 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/03/12 20:31:04 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Fri Mar 12 20:31:04 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sys