使用hive访问elasticsearch的数据

使用hive访问elasticsearch的数据

1.配置
将elasticsearch-hadoop-2.1.1.jar拷贝到hive/lib

hive -hiveconf hive.aux.jars.path=/usr/local/hive-1.2.1/lib/elasticsearch-hadoop-2.1.1.jar

或者配置：
hive-site.xml

<property>
  <name>hive.aux.jars.path</name>
  <value>file:///usr/local/hive-1.2.1/lib/elasticsearch-hadoop-2.1.1.jar</value>
  <description>A comma separated list (with no spaces) of the jar files</description>
</property>

2.创建外表

CREATE EXTERNAL TABLE achi_ex(
vid string,
dtime timestamp,
platid bigint)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘t’
LINES TERMINATED BY ‘n’
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(‘es.resource’ = ‘db_1/achi’,
‘es.index.auto.create’ = ‘true’);

*hive 中的timestamp和ES中的date相互映射

3.导入数据
insert overwrite table achi_ex select * from achi;

4.测试

select * from achi_ex limit 10;

错误：

Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable

hive表中field类型为int时，映射到es中变成long，所以会报此错误。将hive表中int改为bigint即可。

CREATE EXTERNAL TABLE achi_ex(
vid string,
dtime timestamp,
platid bigint)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘t’
LINES TERMINATED BY ‘n’
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(‘es.resource’ = ‘db_1/achi’,
‘es.mapping.names’ = ‘vid:vID,dtime:dTime,platid:PlatID
‘);

最后

以上就是忐忑乐曲为你收集整理的使用hive访问elasticsearch的数据的全部内容，希望文章能够帮你解决使用hive访问elasticsearch的数据所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。