Hive 解决中文分区问题 Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) forHive 中文分区

51 阅读 0 评论 34 点赞

我是靠谱客的博主能干心锁，最近开发中收集的这篇文章主要介绍Hive 解决中文分区问题 Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) forHive 中文分区，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

Hive 中文分区

初学Hive。在创建动态分区的时候。报错。

Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation ‘=’

执行SQL为：

INSERT INTO TABLE test_company PARTITION (p_province) SELECT
    company_id,
    province,
    company_name,
    ... , 
    Clicks,
    province
FROM
    landing.test_company
LIMIT 10;

报错信息为：

java.sql.SQLException: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='

解决问题心历路程：

0. 怀疑是编码问题

改了编码。。但是没改全。。。
结果：依然报相同错误。

1. 怀疑数据源的乱码问题

刚开始使用了两个分区字段，改为了一个。

结果：不是根本问题。依然报错。

2. 使用数字分区

结果：通过。

3. 使用静态分区

执行SQL为：

alter table test_company add partition (p_province='北京市');

结果：依然相同错误

4. 找到改数据库所有编码的脚本。。

结果：问题解决。OVER。

这个神脚本我整理一下：

alter database hive default character set utf8;

alter table BUCKETING_COLS default character set utf8;

alter table CDS default character set utf8;

alter table COLUMNS_V2 default character set utf8;

alter table DATABASE_PARAMS default character set utf8;

alter table DBS default character set utf8;

alter table FUNCS default character set utf8;

alter table FUNC_RU default character set utf8;

alter table GLOBAL_PRIVS default character set utf8;

alter table PARTITIONS default character set utf8;

alter table PARTITION_KEYS default character set utf8;

alter table PARTITION_KEY_VALS default character set utf8;

alter table PARTITION_PARAMS default character set utf8;

alter table PART_COL_STATS default character set utf8;

alter table ROLES default character set utf8;

alter table SDS default character set utf8;

alter table SD_PARAMS default character set utf8;

alter table SEQUENCE_TABLE default character set utf8;

alter table SERDES default character set utf8;

alter table SERDE_PARAMS default character set utf8;

alter table SKEWED_COL_NAMES default character set utf8;

alter table SKEWED_COL_VALUE_LOC_MAP default character set utf8;

alter table SKEWED_STRING_LIST default character set utf8;

alter table SKEWED_STRING_LIST_VALUES default character set utf8;

alter table SKEWED_VALUES default character set utf8;

alter table SORT_COLS default character set utf8;

alter table TABLE_PARAMS default character set utf8;

alter table TAB_COL_STATS default character set utf8;

alter table TBLS default character set utf8;

alter table VERSION default character set utf8;

alter table BUCKETING_COLS convert to character set utf8;

alter table CDS convert to character set utf8;

alter table COLUMNS_V2 convert to character set utf8;

alter table DATABASE_PARAMS convert to character set utf8;

alter table DBS convert to character set utf8;

alter table FUNCS convert to character set utf8;

alter table FUNC_RU convert to character set utf8;

alter table GLOBAL_PRIVS convert to character set utf8;

alter table PARTITIONS convert to character set utf8;

alter table PARTITION_KEYS convert to character set utf8;

alter table PARTITION_KEY_VALS convert to character set utf8;

alter table PARTITION_PARAMS convert to character set utf8;

alter table PART_COL_STATS convert to character set utf8;

alter table ROLES convert to character set utf8;

alter table SDS convert to character set utf8;

alter table SD_PARAMS convert to character set utf8;

alter table SEQUENCE_TABLE convert to character set utf8;

alter table SERDES convert to character set utf8;

alter table SERDE_PARAMS convert to character set utf8;

alter table SKEWED_COL_NAMES convert to character set utf8;

alter table SKEWED_COL_VALUE_LOC_MAP convert to character set utf8;

alter table SKEWED_STRING_LIST convert to character set utf8;

alter table SKEWED_STRING_LIST_VALUES convert to character set utf8;

alter table SKEWED_VALUES convert to character set utf8;

alter table SORT_COLS convert to character set utf8;

alter table TABLE_PARAMS convert to character set utf8;

alter table TAB_COL_STATS convert to character set utf8;

alter table TBLS convert to character set utf8;

alter table VERSION convert to character set utf8;

alter table PART_COL_STATS convert to character set utf8;

SET character_set_client = utf8 ;

SET character_set_connection = utf8 ; 


alter table PART_COL_STATS convert to character set utf8;

SET character_set_database = utf8 ; 

SET character_set_results = utf8 ; 

SET character_set_server = utf8 ; 

SET collation_connection = utf8 ; 

SET collation_database = utf8 ; 

SET collation_server = utf8 ; 

SET NAMES ‘utf8’; 
SET NAMES 'utf8';

Query OK, 0 rows affected (0.00 sec)