mysql与hdfs的数据迁移（sqoop）

1、查看mysql数据

hadoop@ddai-desktop:~$ sqoop list-databases --connect jdbc:mysql://ddai-master:3306/ --username sqoop --password Dai@123456

2、查看mysql表

hadoop@ddai-desktop:~$ sqoop list-tables --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456

3、mysql表导入到hdfs

打开hadoop，用上一篇创建的表

hadoop@ddai-desktop:~$ sqoop import --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --target-dir /user/dept

4、查看hdfs

hadoop@ddai-desktop:~$ hdfs dfs -ls /user/dept

hadoop@ddai-desktop:~$ hdfs dfs -cat /user/dept/part-m-00000

5、清空dept表

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456 
mysql> use sqoop;
mysql> truncate dept;

6、数据从hdfs导出到MySQL表

hadoop@ddai-desktop:~$ sqoop export --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --export-dir /user/dept

7、查询dept表

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 24
Server version: 8.0.26-0ubuntu0.20.04.2 (Ubuntu)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> use sqoop;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from dept;
+------+-----------+
| id   | name      |
+------+-----------+
| 1245 | 云计算    |
| 3512 | 物联网    |
| 3545 | 大数据    |
+------+-----------+
3 rows in set (0.00 sec)

mysql> quit
Bye

8、Sqoop Import 增量导入到 HDFS

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456 
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 25
Server version: 8.0.26-0ubuntu0.20.04.2 (Ubuntu)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> use sqoop;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> insert into dept values(7854,'今天是2021七夕');
Query OK, 1 row affected (0.00 sec)

mysql> select * from dept;
+------+---------------------+
| id   | name                |
+------+---------------------+
| 1245 | 云计算              |
| 3512 | 物联网              |
| 3545 | 大数据              |
| 7854 | 今天是2021七夕      |
+------+---------------------+
4 rows in set (0.00 sec)

mysql> quit
Bye

hadoop@ddai-desktop:~$ sqoop import --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --target-dir /user/dept -incremental append --check-column id

最后

以上就是安详红酒为你收集整理的mysql与hdfs的数据迁移（sqoop）的全部内容，希望文章能够帮你解决mysql与hdfs的数据迁移（sqoop）所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。