1. 首页
  2. 文章中心
  3. ElasticSearch

ES 分词器analyzer设置，包括创建更新索引时和查询时

83 阅读 0 评论
我是靠谱客的博主 无聊西牛，最近开发中收集的这篇文章主要介绍ES 分词器analyzer设置，包括创建更新索引时和查询时，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

analyzer  

分词器使用的两个情形：  
1，Index time analysis.  创建或者更新文档时，会对文档进行分词
2，Search time analysis.  查询时，对查询语句分词

 

- 查询时通过analyzer指定分词器

POST test_index/_search
{
"query": {
"match": {
"name": {
"query": "lin",
"analyzer": "standard"
}
}
}
}

- 创建index mapping时指定search_analyzer

PUT test_index
{
"mappings": {
"doc": {
"properties": {
"title":{
"type": "text",
"analyzer": "whitespace",
"search_analyzer": "standard"
}
}
}
}
}

索引时分词是通过配置 Index mapping中的每个字段的参数analyzer指定的

不指定分词时，会使用默认的standard分词器

注意：

 明确字段是否需要分词，不需要分词的字段将type设置为keyword，可以节省空间和提高写性能。

_analyzer api    

POST _analyze
{
"analyzer": "standard",
"text": "this is a test"
}
# 可以查看text的内容使用standard分词后的结果
{
"tokens":
[
{
"token": "this",
"start_offset": 0,
"end_offset": 4,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "is",
"start_offset": 5,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 1
},
{
"token": "a",
"start_offset": 8,
"end_offset": 9,
"type": "<ALPHANUM>",
"position": 2
},
{
"token": "test",
"start_offset": 10,
"end_offset": 14,
"type": "<ALPHANUM>",
"position": 3
}
]
}

 

设置analyzer

PUT test
{
"settings": {
"analysis": { #自定义分词器
"analyzer": { # 关键字
"my_analyzer":{ # 自定义的分词器
"type":"standard", #分词器类型standard
"stopwords":"_english_" #standard分词器的参数，默认的stopwords是_none_
}
}
}
},
"mappings": {
"doc":{
"properties": {
"my_text":{
"type": "text",
"analyzer": "standard", # my_text字段使用standard分词器
"fields": {
"english":{ # my_text.english字段使用上面自定义的my_analyzer分词器
"type": "text",
"analyzer": "my_analyzer"
}
}
}
}
}
}
}
POST test/_analyze
{
"field": "my_text", # my_text字段使用的是standard分词器
"text": ["The test message."]
}
-------------->[the,test,message]
POST test/_analyze
{
"field": "my_text.english", #my_text.english使用的是my_analyzer分词器
"text": ["The test message."]
}
------------>[test,message]

ES内置了很多种analyzer。比如：

  • standard  由以下组成
    • tokenizer：Standard Tokenizer

token filter：Standard Token Filter，Lower Case Token Filter，Stop Token Filter 

analyzer API测试 ：
POST _analyze
{
"analyzer": "standard",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

得到结果：

{
"tokens": [
{
"token": "the",
"start_offset": 0,
"end_offset": 3,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "2",
"start_offset": 4,
"end_offset": 5,
"type": "<NUM>",
"position": 1
},
{
"token": "quick",
"start_offset": 6,
"end_offset": 11,
"type": "<ALPHANUM>",
"position": 2
},
{
"token": "brown",
"start_offset": 12,
"end_offset": 17,
"type": "<ALPHANUM>",
"position": 3
},
{
"token": "foxes",
"start_offset": 18,
"end_offset": 23,
"type": "<ALPHANUM>",
"position": 4
},
{
"token": "jumped",
"start_offset": 24,
"end_offset": 30,
"type": "<ALPHANUM>",
"position": 5
},
{
"token": "over",
"start_offset": 31,
"end_offset": 35,
"type": "<ALPHANUM>",
"position": 6
},
{
"token": "the",
"start_offset": 36,
"end_offset": 39,
"type": "<ALPHANUM>",
"position": 7
},
{
"token": "lazy",
"start_offset": 40,
"end_offset": 44,
"type": "<ALPHANUM>",
"position": 8
},
{
"token": "dog's",
"start_offset": 45,
"end_offset": 50,
"type": "<ALPHANUM>",
"position": 9
},
{
"token": "bone",
"start_offset": 51,
"end_offset": 55,
"type": "<ALPHANUM>",
"position": 10
}
]
}

whitespace  空格为分隔符

POST _analyze
{
"analyzer": "whitespace",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
-->
[ The,2,QUICK,Brown-Foxes,jumped,over,the,lazy,dog's,bone. ]

simple

POST _analyze
{
"analyzer": "simple",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
---> [ the, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

stop   默认stopwords用_english_ 

POST _analyze
{
"analyzer": "stop",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
-->[ quick, brown, foxes, jumped, over, lazy, dog, s, bone ]
可选参数：
# stopwords
# stopwords_path

keyword  不分词的

POST _analyze
{
"analyzer": "keyword",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."]
}
得到
"token": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone." 一条完整的语句

 

第三方analyzer插件---中文分词（ik分词器）

es内置很多分词器，但是对中文分词并不友好，例如使用standard分词器对一句中文话进行分词，会分成一个字一个字的。这时可以使用第三方的Analyzer插件，比如 ik、pinyin等。这里以ik为例

1，首先安装插件，重启es：

# bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.3.0/elasticsearch-analysis-ik-6.3.0.zip
# /etc/init.d/elasticsearch restart

2，使用示例：

GET _analyze
{
"analyzer": "ik_max_word",
"text": "你好吗?我有一句话要对你说呀。"
}
得到：
{
"tokens": [
{
"token": "你好",
"start_offset": 0,
"end_offset": 2,
"type": "CN_WORD",
"position": 0
},
{
"token": "好吗",
"start_offset": 1,
"end_offset": 3,
"type": "CN_WORD",
"position": 1
},
{
"token": "我",
"start_offset": 4,
"end_offset": 5,
"type": "CN_CHAR",
"position": 2
},
{
"token": "有",
"start_offset": 5,
"end_offset": 6,
"type": "CN_CHAR",
"position": 3
},
{
"token": "一句话",
"start_offset": 6,
"end_offset": 9,
"type": "CN_WORD",
"position": 4
},
{
"token": "一句",
"start_offset": 6,
"end_offset": 8,
"type": "CN_WORD",
"position": 5
},
{
"token": "一",
"start_offset": 6,
"end_offset": 7,
"type": "TYPE_CNUM",
"position": 6
},
{
"token": "句话",
"start_offset": 7,
"end_offset": 9,
"type": "CN_WORD",
"position": 7
},
{
"token": "句",
"start_offset": 7,
"end_offset": 8,
"type": "COUNT",
"position": 8
},
{
"token": "话",
"start_offset": 8,
"end_offset": 9,
"type": "CN_CHAR",
"position": 9
},
{
"token": "要对",
"start_offset": 9,
"end_offset": 11,
"type": "CN_WORD",
"position": 10
},
{
"token": "你",
"start_offset": 11,
"end_offset": 12,
"type": "CN_CHAR",
"position": 11
},
{
"token": "说呀",
"start_offset": 12,
"end_offset": 14,
"type": "CN_WORD",
"position": 12
}
]
}

参考：https://github.com/medcl/elasticsearch-analysis-ik

 

 

还可以用内置的 character filter, tokenizer, token filter 组装一个analyzer(custom analyzer)

custom  定制analyzer，由以下几部分组成

  • 0个或多个character filters
  • 1个tokenizer
  • 0个或多个 token filters
PUT t_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer":{
"type":"custom",
"tokenizer":"standard",
"char_filter":["html_strip"],
"filter":["lowercase"]
}
}
}
}
}
POST t_index/_analyze
{
"analyzer": "my_analyzer",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's <b> bone.</b>"]
}
得到：[the,2,quick,brown,foxes,jumped,over,the,lazy,dog's,bone]

自定义分词器

自定义分词需要在索引的配置中设定，如下所示：

PUT test_index
{
"settings": {
"analysis": { # 分词设置，可以自定义
"char_filter": {}, #char_filter 关键字
"tokenizer": {}, #tokenizer 关键字
"filter": {}, #filter 关键字
"analyzer": {} #analyzer 关键字
}
}
}

character filter  在tokenizer之前对原始文本进行处理，比如增加，删除，替换字符等

会影响后续tokenizer解析的position和offset信息

html strip  除去html标签和转换html实体

  • 参数：escaped_tags不删除的标签
POST _analyze
{
"tokenizer": "keyword",
"char_filter": ["html_strip"],
"text": ["<p>I&apos;m so <b>happy</b>!</p>"]
}
得到：
"token": """
I'm so happy!
"""
#配置示例
PUT t_index
{
"settings": {
"analysis": {
"analyzer": {
#关键字
"my_analyzer":{
#自定义analyzer
"tokenizer":"keyword",
"char_filter":["my_char_filter"]
}
},
"char_filter": {
#关键字
"my_char_filter":{
#自定义char_filter
"type":"html_strip",
"escaped_tags":["b"]
#不从文本中删除的HTML标记数组
}
}
}
}
}
POST t_index/_analyze
{
"analyzer": "my_analyzer",
"text": ["<p>I&apos;m so <b>happy</b>!</p>"]
}
得到：
"token": """
I'm so <b>happy</b>!
""",

mapping    映射类型，以下参数必须二选一

  • mappings 指定一组映射，每个映射格式为 key=>value
  • mappings_path 绝对路径或者相对于config路径   key=>value
PUT t_index
{
"settings": {
"analysis": {
"analyzer": { #关键字
"my_analyzer":{ #自定义分词器
"tokenizer":"standard",
"char_filter":"my_char_filter"
}
},
"char_filter": { #关键字
"my_char_filter":{  #自定义char_filter
"type":"mapping",
"mappings":[ #指明映射关系
":)=>happy",
":(=>sad"
]
}
}
}
}
}
POST t_index/_analyze
{
"analyzer": "my_analyzer",
"text": ["i am so :)"]
}
得到 [i,am,so,happy]

pattern replace

  • pattern参数  正则
  • replacement 替换字符串 可以使用$1..$9
  • flags  正则标志

tokenizer  将原始文档按照一定规则切分为单词

  • standard
    • 参数：max_token_length，最大token长度，默认是255
PUT t_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer":{
"tokenizer":"my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer":{
"type":"standard",
"max_token_length":5
}
}
}
}
}
POST t_index/_analyze
{
"analyzer": "my_analyzer",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."]
}
得到 [ The, 2, QUICK, Brown, Foxes, jumpe, d, over, the, lazy, dog's, bone ]
# jumped 长度为6 在5这个位置被分割

letter    非字母时分成多个terms

POST _analyze
{
"tokenizer": "letter",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."]
}
得到 [ The, QUICK, Brown, Foxes, jumped, over, the, lazy, dog, s, bone ]

lowcase  跟letter tokenizer一样 ，同时将字母转化成小写

POST _analyze
{
"tokenizer": "lowercase",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
得到
[ the, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

whitespace   按照空白字符分成多个terms

  • 参数：max_token_length
POST _analyze
{
"tokenizer": "whitespace",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
得到 [ The, 2, QUICK, Brown-Foxes, jumped, over, the, lazy, dog's, bone. ]

keyword   空操作，输出完全相同的文本

  • 参数：buffer_size，单词一个term读入缓冲区的长度，默认256
POST _analyze
{
"tokenizer": "keyword",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."]
}
得到"token": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone." 一个完整的文本

token filter   针对tokenizer 输出的单词进行增删改等操作

lowercase  将输出的单词转化成小写

POST _analyze
{
"filter": ["lowercase"],
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone"]
}
--->
"token": "the 2 quick brown-foxes jumped over the lazy dog's bone"
PUT t_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":"lowercase"
}
}
}
}
}
POST t_index/_analyze
{
　　"analyzer": "my_analyzer",
"text": ["The 2 QUICK Brown-Foxes jumped over the lazy dog's bone"]
}

stop  从token流中删除stop words 。

参数有：

# stopwords
要使用的stopwords， 默认_english_
# stopwords_path
# ignore_case
设置为true则为小写，默认false
# remove_trailing
PUT t_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":"my_filter"
}
},
"filter": {
"my_filter":{
"type":"stop",
"stopwords":["and","or","not"]
}
}
}
}
}
POST t_index/_analyze
{
"analyzer": "my_analyzer",
"text": ["lucky and happy not sad"]
}
-------------->
[lucky,happy,sad]

 

最后

以上就是无聊西牛为你收集整理的ES 分词器analyzer设置，包括创建更新索引时和查询时的全部内容，希望文章能够帮你解决ES 分词器analyzer设置，包括创建更新索引时和查询时所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。

本图文内容来源于网友提供，作为学习参考使用，或来自网络收集整理，版权属于原作者所有。
点赞(55)

相关文章

lucene索引时analyzer的使用
lucene索引时analyzer的使用
Elasticsearch 自定义过滤器示例
Elasticsearch 自定义过滤器示例
es 创建索引报错_ES添加映射报错analyzer [ik_max_word] not found for field [title]
es 创建索引报错_ES添加映射报错analyzer [ik_max_word] not found for field [title]
elasticsearch在不删除索引情况下，新增tpye的字段
elasticsearch在不删除索引情况下，新增tpye的字段
ES 分词器analyzer设置，包括创建更新索引时和查询时
ES 分词器analyzer设置，包括创建更新索引时和查询时
es学习-索引配置
es学习-索引配置
Elasticsearch Mapping 中配置自定义Analyzer
Elasticsearch Mapping 中配置自定义Analyzer
Oracle建完索引后要analyze一下才行!
Oracle建完索引后要analyze一下才行!

评论列表共有 0 条评论

立即
投稿
返回
顶部