【Elasticsearch】用ElasticSearch实现基于标签的兴趣推荐

￡神魔★判官ぃ 2022-10-10 04:58 196阅读 0赞

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMzgzNDM1_size_16_color_FFFFFF_t_70_pic_center]

# 1.概述 #

转载：[https://my.oschina.net/waterbear/blog/4493422][https_my.oschina.net_waterbear_blog_4493422]

主要是为了实操一遍，同时更改curl的方式。

# 前提 #

操作系统：mac  
ElasticSearch 7.8

# 推荐原理 #

*  文章索引中有字段tags，存储了文章有关的标签
 *  每个用户都有自己的兴趣标签tags
 *  兴趣推荐就是用兴趣标签去匹配文章的标签，用户的一个兴趣标签命中N篇文章，用户的多个兴趣标签命中M篇文章，M和N有交叉，即文章中有重复，重复出现次数最多的文章就是最贴近用户兴趣的。原理理解起来简单，使用ES的目的是解决快速查询和排序的问题。

# 创建索引 #

PUT rcmd
    
    PUT /rcmd/_mapping
    { 
      "properties": { 
        "tags": { 
          "type": "keyword",
          "store": true
        },
        "update_time": { 
          "type": "date",
          "store": true
        }
      }
    }

两个字段：

*  tags，文章的兴趣标签，keyword类型就是不需要全文检索，标签以数组的形式存放
 *  update\_time，更新时间，这是给兴趣推荐加一个额外的排序条件，实际项目中往往是需要结合时间和匹配度来排序的

# 模拟数据 #

POST /rcmd/_doc 
    { 
     	"tags": [
     		"布料",
     		"抹布",
     		"裤子",
     		"衣服",
     		"生活"
     	],
     	"update_time": "2020-06-01T00:02:11.030"
     }
     
     # 再插入一条，同样标签，但是时间不一样，后面例子中有妙用
     
    POST /rcmd/_doc
     { 
     	"tags": [
     		"布料",
     		"抹布",
     		"裤子",
     		"衣服",
     		"生活"
     	],
     	"update_time": "2020-07-01T00:02:11.030"
     }
     
     
    GET /rcmd/_search
    
    POST /rcmd/_doc
    { 
      "tags": [
        "啤酒",
        "米酒",
        "饮料",
        "餐饮",
        "生活"
      ],
      "update_time": "2020-06-02T00:02:11.030"
    }
    
    POST /rcmd/_doc
    { 
     	"tags": [
     		"火锅",
     		"自助餐",
     		"外卖",
     		"烧烤",
     		"餐饮"
     	],
     	"update_time": "2020-06-03T00:02:11.030"
     }
     
     
     
    POST /rcmd/_doc
    { 
     	"tags": [
     		"太阳",
     		"月亮",
     		"大海",
     		"星星",
     		"自然"
     	],
     	"update_time": "2020-06-01T00:02:11.030"
     }
     
    POST /rcmd/_doc
    { 
     	"tags": [
     		"人类",
     		"动物",
     		"植物",
     		"地球",
     		"自然"
     	],
     	"update_time": "2020-06-01T00:02:11.030"
     }
     
    POST /rcmd/_doc
    { 
     	"tags": [
     		"男人",
     		"女人",
     		"小孩",
     		"老人",
     		"人类"
     	],
     	"update_time": "2020-06-02T00:02:11.030"
     }

最终数据如下

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMzgzNDM1_size_16_color_FFFFFF_t_70]

# 固定分数查询 #

GET /rcmd/_search 
     { 
     	"query": { 
     		"bool": { 
     			"should": [
     				{ 
     					"constant_score": { 
     						"boost": 1,
     						"filter": { 
     							"match": { 
     								"tags": "生活"
     							}
     						}
     					}
     				},
     				{ 
     					"constant_score": { 
     						"boost": 1,
     						"filter": { 
     							"match": { 
     								"tags": "衣服"
     							}
     						}
     					}
     				},
     				{ 
     					"constant_score": { 
     						"boost": 1,
     						"filter": { 
     							"match": { 
     								"tags": "火锅"
     							}
     						}
     					}
     				}
     			]
     		}
     	}
     }

should表达式的意义是匹配“生活”、“衣服”、“火锅”三个标签中任何一个的文章都可以返回。用constant\_score查询，如果某个文章涵盖标签越多分值就越高。也就是说如果某个文章标签完全涵盖了这三个标签，那么它的分值最高的。查询结果如下：

{ 
      "took": 2,
      "timed_out": false,
      "_shards": { 
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": { 
        "total": { 
          "value": 4,
          "relation": "eq"
        },
        "max_score": 2.0,
        "hits": [
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "brQO63MBTdXKc2eArv9A",
            "_score": 2.0,
            "_source": { 
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ],
              "update_time": "2020-06-01T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "b7QP63MBTdXKc2eAPf_Y",
            "_score": 2.0,
            "_source": { 
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ],
              "update_time": "2020-07-01T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cLQQ63MBTdXKc2eA6_8v",
            "_score": 1.0,
            "_source": { 
              "tags": [
                "啤酒",
                "米酒",
                "饮料",
                "餐饮",
                "生活"
              ],
              "update_time": "2020-06-02T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cbQS63MBTdXKc2eAcP-N",
            "_score": 1.0,
            "_source": { 
              "tags": [
                "火锅",
                "自助餐",
                "外卖",
                "烧烤",
                "餐饮"
              ],
              "update_time": "2020-06-03T00:02:11.030"
            }
          }
        ]
      }
    }

有两篇文章涵盖了其中两个标签“生活”和“衣服”，得分为2，排到了前面。这个排序基本满足了兴趣匹配的要求。

# 兴趣标签权值 #

实际的项目中往往是用户的兴趣标签的权值不一样，假设用户的兴趣标签是\[“火锅”,“生活”,“衣服”\]，排在越前面的权重越高，查询的时候需要给关键词设定权重，上面的查询语句所有boost都是默认值1，现在根据需求改动权值再查询。

GET /rcmd/_search 
    { 
     	"query": { 
     		"bool": { 
     			"should": [
     				{ 
     					"constant_score": { 
     						"boost": 1,
     						"filter": { 
     							"match": { 
     								"tags": "生活"
     							}
     						}
     					}
     				},
     				{ 
     					"constant_score": { 
     						"boost": 4,
     						"filter": { 
     							"match": { 
     								"tags": "衣服"
     							}
     						}
     					}
     				},
     				{ 
     					"constant_score": { 
     						"boost": 6,
     						"filter": { 
     							"match": { 
     								"tags": "火锅"
     							}
     						}
     					}
     				}
     			]
     		}
     	}
     }

分别给三个词加上权重6、4、1，查询结果如下：

{ 
      "took": 1,
      "timed_out": false,
      "_shards": { 
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": { 
        "total": { 
          "value": 4,
          "relation": "eq"
        },
        "max_score": 6.0,
        "hits": [
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cbQS63MBTdXKc2eAcP-N",
            "_score": 6.0,
            "_source": { 
              "tags": [
                "火锅",
                "自助餐",
                "外卖",
                "烧烤",
                "餐饮"
              ],
              "update_time": "2020-06-03T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "brQO63MBTdXKc2eArv9A",
            "_score": 5.0,
            "_source": { 
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ],
              "update_time": "2020-06-01T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "b7QP63MBTdXKc2eAPf_Y",
            "_score": 5.0,
            "_source": { 
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ],
              "update_time": "2020-07-01T00:02:11.030"
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cLQQ63MBTdXKc2eA6_8v",
            "_score": 1.0,
            "_source": { 
              "tags": [
                "啤酒",
                "米酒",
                "饮料",
                "餐饮",
                "生活"
              ],
              "update_time": "2020-06-02T00:02:11.030"
            }
          }
        ]
      }
    }

可以看到包含“火锅”的文章排到了第一，包含“衣服”和“生活”的文章虽然两个词都命中，但是在权值的弱化之下排到了第二第三位。

# 多条件排序 #

GET /rcmd/_search
    { 
      "query": { 
        "function_score": { 
          "query": { 
            "bool": { 
              "must": [
                { 
                  "range": { 
                    "update_time": { 
                      "from": "2020-06-01",
                      "to": "2020-08-01"
                    }
                  }
                },
                { 
                  "bool": { 
                    "should": [
                      { 
                        "term": { 
                          "tags": { 
                            "term": "火锅",
                            "boost": 2
                          }
                        }
                      },
                      { 
                        "term": { 
                          "tags": { 
                            "term": "衣服",
                            "boost": 1
                          }
                        }
                      },
                      { 
                        "term": { 
                          "tags": { 
                            "term": "生活",
                            "boost": 1
                          }
                        }
                      }
                    ]
                  }
                }
              ]
            }
          },
          "functions": [
            { 
              "gauss": { 
                "update_time": { 
                  "scale": "3d",
                  "origin": "2020-07-02T00:01:00.000"
                }
              }
            }
          ]
        }
      },
      "_source": { 
        "include": [
          "tags",
          "update_time"
        ]
      },
      "from": 0,
      "size": 10
    }

以上是相对完整的一个查询，首先对update\_time发布时间做了限制，只选择一定范围内的数据，随后是标签的匹配，多个标签匹配条件之间是"OR"的关系，标签具有不同的权重，接下来用衰减函数gauss对update\_time做衰减排序，衰减函数的意义是越近越好，scale": "3d"就是以3天为一个阶梯先对数据进行排序，相同阶梯内的数据再按照标签匹配度排序。 注：gauss中的origin可以不指定 最终的查询结果：

{ 
      "took": 2,
      "timed_out": false,
      "_shards": { 
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": { 
        "total": { 
          "value": 4,
          "relation": "eq"
        },
        "max_score": 3.6649413,
        "hits": [
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "b7QP63MBTdXKc2eAPf_Y",
            "_score": 3.6649413,
            "_source": { 
              "update_time": "2020-07-01T00:02:11.030",
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ]
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cbQS63MBTdXKc2eAcP-N",
            "_score": 4.4511746E-28,
            "_source": { 
              "update_time": "2020-06-03T00:02:11.030",
              "tags": [
                "火锅",
                "自助餐",
                "外卖",
                "烧烤",
                "餐饮"
              ]
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "cLQQ63MBTdXKc2eA6_8v",
            "_score": 1.764942E-30,
            "_source": { 
              "update_time": "2020-06-02T00:02:11.030",
              "tags": [
                "啤酒",
                "米酒",
                "饮料",
                "餐饮",
                "生活"
              ]
            }
          },
          { 
            "_index": "rcmd",
            "_type": "_doc",
            "_id": "brQO63MBTdXKc2eArv9A",
            "_score": 2.8566082E-32,
            "_source": { 
              "update_time": "2020-06-01T00:02:11.030",
              "tags": [
                "布料",
                "抹布",
                "裤子",
                "衣服",
                "生活"
              ]
            }
          }
        ]
      }
    }

同样是匹配了“衣服”和“生活”的两篇文章，一篇在最前面，一篇在最后面，是因为update\_time的缘故，一篇是7月1日发布的，另一篇在6月1日，不在同一时间阶梯内，日期久远的排到了后面。中间的两篇，各自匹配了一个标签，分别是“烧烤”和“生活”，两篇文章时间阶梯没有明显的区别，然而匹配“火锅”的排到了前面，是因为“火锅”的关键词加了较高的权重。 至此，我们实现了按照标签匹配文章，并且结合了时间因素和匹配度评分的兴趣推荐。

# 后续问题 #

*  本文仅仅实现了`推荐中信息的匹配和排序`，实际的推荐系统中还有信息流中过滤已推文章的问题，取备选文章和历史文章的交集是比较耗时的运算；此外还要解决`用户Feed流推拉问题`。
 *  在本文中没有提及`用户兴趣标签的累积操作`，`理论上来说只要用户读了相应的文章，就根据文章的标签给用户兴趣标签累计加分，实际项目中往往需要处理兴趣标签截断统一降权`，`例如标签库有1万个标签，用户进行多次阅读行为之后，一个用户和1万个标签都会有关系，分值大小不同的区别而已，如果一直根据高分的兴趣标签给用户推荐文章，兴趣标签就形成了马太效应，新的兴趣标签没有机会超过累积高分的标签，兴趣推送会越来越窄，这时候就需要截取一定数量的兴趣标签，例如截取前100个，然后统一降分，给用户的新兴趣超赶机会。`
 *  以上例子没有在超大数据环境下测试过，还没有具体的性能指标。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMzgzNDM1_size_16_color_FFFFFF_t_70_pic_center]: /images/20221005/57e86451576e4fe3a569627f511d1b20.png
[https_my.oschina.net_waterbear_blog_4493422]: https://my.oschina.net/waterbear/blog/4493422
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMzgzNDM1_size_16_color_FFFFFF_t_70]: /images/20221005/a713da48eebf495ea4c4cf43098f421b.png