lucene 3.6 Similarity doc

╰+哭是因爲堅強的太久メ 2022-06-12 11:39 159阅读 0赞

Expert: Scoring API.

Similarity defines the components of Lucene scoring. Overriding computation of these components is a convenient way to alter Lucene scoring.

\[翻译\]Similarity定义了Lucene评分的组件，覆盖这个组件的计算方法是一个Lucene评分机制很好的办法

Suggested reading: [ Introduction To Information Retrieval, Chapter 6][Introduction To Information Retrieval_ Chapter 6].

\[翻译\]建议阅读：

The following describes how Lucene scoring evolves from underlying information retrieval models to (efficient) implementation. We first brief on*VSM Score*, then derive from it *Lucene's Conceptual Scoring Formula*, from which, finally, evolves*Lucene's Practical Scoring Function* (the latter is connected directly with Lucene classes and methods).

\[翻译\]以下描述了Lucene如何将潜在的信息检索模型演化到了高效的实现中。我们首先简单介绍下VSM评分机制，然后从它演化到Lucene的概念评分公式，并最终演化到Lucene的实践评分函数（后者直接关系到Lucene的class和方法）

Lucene combines [ Boolean model (BM) of Information Retrieval][Boolean model _BM_ of Information Retrieval] with [ Vector Space Model (VSM) of Information Retrieval][Vector Space Model _VSM_ of Information Retrieval] \- documents "approved" by BM are scored by VSM.

\[翻译\]Lucene结合了布尔模型和VSM模型。过滤文档由BM完成，评分又VSM完成

In VSM, documents and queries are represented as weighted vectors in a multi-dimensional space, where each distinct index term is a dimension, and weights are[Tf-idf][] values.

\[翻译\]在VSM中，文档和查询表达成多维空间中的加权向量，每一个索引项（一般就是一个词）都是一个维度，权重由tf-idf计算得到

VSM does not require weights to be *Tf-idf* values, but *Tf-idf* values are believed to produce search results of high quality, and so Lucene is using*Tf-idf*. *Tf* and *Idf* are described in more detail below, but for now, for completion, let's just say that for given term*t* and document (or query) *x*, *Tf(t,x)* varies with the number of occurrences of term*t* in *x* (when one increases so does the other) and *idf(t)* similarly varies with the inverse of the number of index documents containing term*t*.

\[翻译\]VSM并不需要权重一定由tf-idf计算得到，但是tf-idf值一般被认为能够产生高质量的搜索结果，因此Lucene使用了tf-idf。tf和idf在下面将作详细描述，但是现在为了完整性，让我们首先明确，对于给定的词项t和文档(或者查询)x，tf(t,x)会随着t在x中出现的数量而变化，idf(t)会随着包含词项t的文档数量的倒数变化。

*VSM score* of document *d* for query *q* is the [ Cosine Similarity][Cosine Similarity] of the weighted query vectors *V(q)* and *V(d)*:

\[翻译\]文档d对于查询q的VSM分数是V（q）和V(d)的向量的余弦相似度：

<table style="width:auto"> 
 <tbody> 
  <tr> 
   <td> 
    <table> 
     <tbody> 
      <tr> 
       <td> 
        <table> 
         <tbody> 
          <tr> 
           <td align="right">cosine-similarity(q,d) &nbsp; = &nbsp;</td> 
           <td align="center"> 
            <table> 
             <tbody> 
              <tr> 
               <td align="center" style="text-align:center"><small>V(q)&nbsp;·&nbsp;V(d)</small></td> 
              </tr> 
              <tr> 
               <td align="center" style="text-align:center">–––––––––</td> 
              </tr> 
              <tr> 
               <td align="center" style="text-align:center"><small>|V(q)|&nbsp;|V(d)|</small></td> 
              </tr> 
             </tbody> 
            </table> </td> 
          </tr> 
         </tbody> 
        </table> </td> 
      </tr> 
     </tbody> 
    </table> </td> 
  </tr> 
  <tr> 
   <td>  <span style=""><u>VSM Score</u></span>  </td> 
  </tr> 
 </tbody> 
</table>

Where *V(q)* · *V(d)* is the [ dot product][dot product] of the weighted vectors, and *|V(q)|* and *|V(d)|* are their[Euclidean norms][].

\[翻译\]这里V(q)\*V(d)是两个向量的点积。然后 *|V(q)|* 和*|V(d)|*是它们的欧式距离。

Note: the above equation can be viewed as the dot product of the normalized weighted vectors, in the sense that dividing*V(q)* by its euclidean norm is normalizing it to a unit vector.

\[翻译\]注意：上面的等式可以当作两个向量的标准化点积。

Lucene refines *VSM score* for both search quality and usability:

\[翻译\]Lucene在搜索质量和实用性上优化了VSM评分法

*  Normalizing *V(d)* to the unit vector is known to be problematic in that it removes all document length information. For some documents removing this info is probably ok, e.g. a document made by duplicating a certain paragraph*10* times, especially if that paragraph is made of distinct terms. But for a document which contains no duplicated paragraphs, this might be wrong. To avoid this problem, a different document length normalization factor is used, which normalizes to a vector equal to or larger than the unit vector: *doc-len-norm(d)*.
 *  \[翻译\]将V(d)标准化成单位向量被认为是有问题的，因为它去掉了所有文档的长度信息。对于一些场景去掉长度是可以的，例如一个文档中包含了一个被复制了10遍的段落，尤其是在那段落是由独特的词项组成。但是对于那些不包含重复段落的文档，这样做是不对的。为了避免这个问题，这里采用一个不同的文档长度标准化方法，这种方法将文档标准化成一个长于单位向量的向量：doc-len-norm(d)
 *  At indexing, users can specify that certain documents are more important than others, by assigning a document boost. For this, the score of each document is also multiplied by its boost value*doc-boost(d)*.
 *  \[翻译\]在索引阶段，用户可以描述某些重要性超过其他的文档，通过赋予一个文档加权。这样的话，每个文档的分数可以乘以它的加权分数 
 *  Lucene is field based, hence each query term applies to a single field, document length normalization is by the length of the certain field, and in addition to document boost there are also document fields boosts.
 *  \[翻译\]LUcene是基于field的，因此每一个查询词项被应用到了一个单个的field上，因此文档的标准化会作用到每一个特定的field上，所以除了有文档的加权，然后各个field的加权
 *  The same field can be added to a document during indexing several times, and so the boost of that field is the multiplication of the boosts of the separate additions (or parts) of that field within the document.
 *  \[翻译\]在索引多次后，同样的field可以被添加到一个文档上，所以那个field的加权是各个部分的加权的和。 
 *  At search time users can specify boosts to each query, sub-query, and each query term, hence the contribution of a query term to the score of a document is multiplied by the boost of that query term*query-boost(q)*.
 *  \[翻译\]在搜索时段用户可以描述一个查询的加权，因此每一个查询词对文档的贡献也会乘以查询词的加权 
 *  A document may match a multi term query without containing all the terms of that query (this is correct for some of the queries), and users can further reward documents matching more query terms through a coordination factor, which is usually larger when more terms are matched: *coord-factor(q,d)*.
 *  \[翻译\]一个文档可以匹配多词项查询，而不需要包含查询的所有词项（这对于某些查询是正确的），用户可以通过coordination factor进一步收获那些匹配了多个查询词的文档，当很多的词匹配后这个factor会很大

Under the simplifying assumption of a single field in the index, we get *Lucene's Conceptual scoring formula*:

\[翻译\]基于单field索引的假设下，Lucene的概念评分公式如下：

<table style="width:auto"> 
 <tbody> 
  <tr> 
   <td> 
    <table> 
     <tbody> 
      <tr> 
       <td> 
        <table> 
         <tbody> 
          <tr> 
           <td align="right">score(q,d) &nbsp; = &nbsp; <span style="color:#FF9933"> coord-factor(q,d)</span> · &nbsp; <span style="color:#CCCC00">query-boost(q)</span> · &nbsp;</td> 
           <td align="center"> 
            <table> 
             <tbody> 
              <tr> 
               <td align="center" style="text-align:center"><small><span style="color:#993399">V(q)&nbsp;·&nbsp;V(d)</span></small></td> 
              </tr> 
              <tr> 
               <td align="center" style="text-align:center">–––––––––</td> 
              </tr> 
              <tr> 
               <td align="center" style="text-align:center"><small><span style="color:#FF33CC">|V(q)|</span></small></td> 
              </tr> 
             </tbody> 
            </table> </td> 
           <td align="right">&nbsp; · &nbsp; <span style="color:#3399FF">doc-len-norm(d)</span> &nbsp; · &nbsp;<span style="color:#3399FF">doc-boost(d)</span></td> 
          </tr> 
         </tbody> 
        </table> </td> 
      </tr> 
     </tbody> 
    </table> </td> 
  </tr> 
  <tr> 
   <td>  <span style=""><u>Lucene Conceptual Scoring Formula</u></span>  </td> 
  </tr> 
 </tbody> 
</table>

The conceptual formula is a simplification in the sense that (1) terms and documents are fielded and (2) boosts are usually per query term rather than per query.

\[翻译\]概念公式是一个简化。（1）词项和文档都field化了（2）加权通常是针对查询词项而不是整个查询

We now describe how Lucene implements this conceptual scoring formula, and derive from it*Lucene's Practical Scoring Function*.

\[翻译\]我们现在来描述Lucene如何实现了这个概念评分公式，然后将它演化到Lucene的实践评分函数中

For efficient score computation some scoring components are computed and aggregated in advance:

\[翻译\]为了效率考虑，一个计算过程将提前完成

*  *Query-boost* for the query (actually for each query term) is known when search starts.
 *  \[翻译\]每个查询的加权（实际上是查询词项）在搜索开始时已经知道 
 *  Query Euclidean norm *|V(q)|* can be computed when search starts, as it is independent of the document being scored. From search optimization perspective, it is a valid question why bother to normalize the query at all, because all scored documents will be multiplied by the same *|V(q)|*, and hence documents ranks (their order by score) will not be affected by this normalization. There are two good reasons to keep this normalization:
 *  \[翻译\]查询的欧式正则|V(q)|可以在搜索开始时得到，因为它与文档的计算过程独立。从搜索优化的角度看，忍受标准化查询的原因是一个很好的问题。因为所有的文档将乘以同一个|V(q)|，于是文档排名并不会因此而受到影响。然而，这里有两个原因去保留这个标准化过程：  
    
    
     *  Recall that [Cosine Similarity][] can be used find how similar two documents are. One can use Lucene for e.g. clustering, and use a document as a query to compute its similarity to other documents. In this use case it is important that the score of document *d3* for query*d1* is comparable to the score of document *d3* for query *d2*. In other words, scores of a document for two distinct queries should be comparable. There are other applications that may require this. And this is exactly what normalizing the query vector *V(q)* provides: comparability (to a certain extent) of two or more queries.
     *  \[翻译\]首先回顾下余弦相似度可以用于表示两个文档的相似度。于是Lucene可以被用于其他地方，如聚类，然后使用文档作为查询来计算它与其他文档的相似度。在这个用例中，文档d3对查询d1的分数与文档d3对查询d2的分数之间的比较就变得有价值了。换句话说，一个文档对于不同的查询的分数要具有区分性。这里还有其他需要用到这个的例子。这就是标准化查询向量V(q)所带来的：两个或者更多的查询间可以比较了
     *  Applying query normalization on the scores helps to keep the scores around the unit vector, hence preventing loss of score data because of floating point precision limitations.
     *  \[翻译\]将查询标准化应用到分数上帮助分数处于单位向量附近，因此它防止了由于浮点数导致的得分损失 
 *  Document length norm *doc-len-norm(d)* and document boost *doc-boost(d)* are known at indexing time. They are computed in advance and their multiplication is saved as a single value in the index:*norm(d)*. (In the equations below, *norm(t in d)* means *norm(field(t) in doc d)* where*field(t)* is the field associated with term *t*.)
 *  \[翻译\]文档长度标准化doc-len-norm(d)和文档加权doc-boost(d)在索引阶段已经知道。它们可以提前计算出来，并保存在索引中作为一个值norm(d)。下面的公式中，norm(t in d)指的是norm(field(t) in doc d)，其中的field(t)与term t相关

*Lucene's Practical Scoring Function* is derived from the above. The color codes demonstrate how it relates to those of the*conceptual* formula:

\[翻译\]Lucene的实践计算函数从上面的总结中得出了！

<table style="width:auto"> 
 <tbody> 
  <tr> 
   <td> 
    <table> 
     <tbody> 
      <tr> 
       <td> 
        <table> 
         <tbody> 
          <tr> 
           <td align="right">score(q,d) &nbsp; = &nbsp; <span style="color:#FF9933">coord(q,d)</span> &nbsp;·&nbsp; <span style="color:#FF33CC">queryNorm(q)</span> &nbsp;·&nbsp;</td> 
           <td align="center" style="text-align:center">∑</td> 
           <td align="right">( <span style="color:#993399">tf(t in d)</span> &nbsp;·&nbsp; <span style="color:#993399">idf(t)</span><sup>2</sup> &nbsp;·&nbsp; <span style="color:#CCCC00">t.getBoost()</span>&nbsp;·&nbsp; <span style="color:#3399FF">norm(t,d)</span> )</td> 
          </tr> 
          <tr> 
           <td>&nbsp;</td> 
           <td align="center" style="text-align:center"><small>t in q</small></td> 
           <td>&nbsp;</td> 
          </tr> 
         </tbody> 
        </table> </td> 
      </tr> 
     </tbody> 
    </table> </td> 
  </tr> 
  <tr> 
   <td>  <span style=""><u>Lucene Practical Scoring Function</u></span>  </td> 
  </tr> 
 </tbody> 
</table>

where

1.  ***tf(t in d)*** correlates to the term's*frequency*, defined as the number of times term *t* appears in the currently scored document*d*. Documents that have more occurrences of a given term receive a higher score. Note that*tf(t in q)* is assumed to be *1* and therefore it does not appear in this equation, However if a query contains twice the same term, there will be two term-queries with that same term and hence the computation would still be correct (although not very efficient). The default computation for *tf(t in d)* in `DefaultSimilarity` is: \[翻译\]ft(t in d)代表了词项的频率，词项t在当前评分的文档d中出现的次数。那些包含给定的词项的数量多的文档将得到高分。注意tf(t in q)被假设为1，因此它不会出现在等式中，然而如果查询中包含同样的词项，那么这个词项将被算两次（当然了，效率不够高）。默认的计算方法为：  
      
    
    
    <table style="width:auto"> 
     <tbody> 
      <tr> 
       <td align="right"><code>tf(t in d)</code> &nbsp; = &nbsp;</td> 
       <td align="center">frequency<sup>½</sup></td> 
      </tr> 
     </tbody> 
    </table>
    
      
      
    
2.  ***idf(t)*** stands for Inverse Document Frequency. This value correlates to the inverse of*docFreq* (the number of documents in which the term *t* appears). This means rarer terms give higher contribution to the total score.*idf(t)* appears for *t* in both the query and the document, hence it is squared in the equation. The default computation for*idf(t)* in `DefaultSimilarity` is: \[翻译\]idf(t)代表了逆文档频率。这个值与docFreq的倒数相关。这意味着某个词出现的文档数越少越好。idf(t)出现在查询和文档两个地方，因此它被做了平方。默认的计算方法是：  
      
    
    
    <table style="width:auto"> 
     <tbody> 
      <tr> 
       <td align="right"><code>idf(t)</code>&nbsp; = &nbsp;</td> 
       <td align="center">1 + log (</td> 
       <td align="center"> 
        <table> 
         <tbody> 
          <tr> 
           <td align="center" style="text-align:center"><small>numDocs</small></td> 
          </tr> 
          <tr> 
           <td align="center" style="text-align:center">–––––––––</td> 
          </tr> 
          <tr> 
           <td align="center" style="text-align:center"><small>docFreq+1</small></td> 
          </tr> 
         </tbody> 
        </table> </td> 
       <td align="center">)</td> 
      </tr> 
     </tbody> 
    </table>
    
      
      
    
3.  ***coord(q,d)*** is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in `coord(q,d)` by the Similarity in effect at search time. \[翻译\]coord(q,d)是一个分数因子，它基于在给定文档中找到的查询词的数目。通常来说，一个包含更多查询词的文档将得到更高的分数。这是一个搜索时产生的因子。  
      
    
4.  ***queryNorm(q)***is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation in`DefaultSimilarity` produces a [ Euclidean norm][Euclidean norms]: \[翻译\]queryNorm(q)是一个标准化因子，为了使不同间的查询比较变得有效。这个因此不会影响文档排序（因为所有的文档都乘以这个因子），只是为了不同查询间的分数不同。这是一个搜索时产生的因子。默认的计算方法如下：  
      
    
    
    <table style="width:auto"> 
     <tbody> 
      <tr> 
       <td align="right">queryNorm(q) &nbsp; = &nbsp; <code>queryNorm(sumOfSquaredWeights)</code> &nbsp; = &nbsp;</td> 
       <td align="center"> 
        <table> 
         <tbody> 
          <tr> 
           <td align="center" style="text-align:center">1</td> 
          </tr> 
          <tr> 
           <td align="center" style="text-align:center">–––––––––––––– </td> 
          </tr> 
          <tr> 
           <td align="center" style="text-align:center">sumOfSquaredWeights<sup>½</sup></td> 
          </tr> 
         </tbody> 
        </table> </td> 
      </tr> 
     </tbody> 
    </table>
    
      
      
    The sum of squared weights (of the query terms) is computed by the query  `Weight` object. For example, a  `BooleanQuery` computes this value as:  \[翻译\]权重的平方和由Weight对象计算得到  
      
    
    
    <table style="width:auto"> 
     <tbody> 
      <tr> 
       <td align="right"><code>sumOfSquaredWeights</code> &nbsp; = &nbsp;<code>q.getBoost()</code> <sup>2</sup> &nbsp;·&nbsp;</td> 
       <td align="center" style="text-align:center">∑</td> 
       <td align="right">( idf(t) &nbsp;·&nbsp; t.getBoost() ) <sup>2</sup> </td> 
      </tr> 
      <tr> 
       <td>&nbsp;</td> 
       <td align="center" style="text-align:center"><small>t in q</small></td> 
       <td>&nbsp;</td> 
      </tr> 
     </tbody> 
    </table>
    
      
      
    
5.  ***t.getBoost()*** is a search time boost of term*t* in the query *q* as specified in the query text (see query syntax), or as set by application calls to `setBoost()`. Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi`TermQuery` objects, and so the boost of a term in the query is accessible by calling the sub-query`getBoost()`. \[翻译\]t.getBoost()是一个查询q中词项t的搜索时加权因子，它应该出现在查询文字中，或者由应用调用setBoost（）得到。注意……  
      
    
6.  ***norm(t,d)*** encapsulates a few (indexing time) boost and length factors:\[翻译\]norm(t,d)封装了一系列在索引是得到的加权和长度因子
    
     *  **Document boost** \- set by calling `doc.setBoost()` before adding the document to the index.\[翻译\]文档加权：在将文档加入索引时调用doc.setBoost()
     *  **Field boost** \- set by calling `field.setBoost()` before adding the field to a document.\[翻译\]域加权：在将field加入文档前调用field.setBoost()
     *  **lengthNorm** \- computed when the document is added to the index in accordance with the number of tokens of this field in the document, so that shorter fields contribute more to the score. LengthNorm is computed by the Similarity class in effect at indexing.\[翻译\]长度标准化：在文档加入索引时计算。它与词在该field中出现的次数有关，所以短field的分更高。长度标准化在索引时计算
    
    The `computeNorm(java.lang.String, org.apache.lucene.index.FieldInvertState)` method is responsible for combining all of these factors into a single float.
    
    When a document is added to the index, all the above factors are multiplied. If the document has multiple fields with the same name, all their boosts are multiplied together: \[翻译\]当以个文档加入索引时，上面提到的所有因此将一起计算，如果有多个field,那么它们将一起连乘加权  
      
    
    
    <table style="width:auto"> 
     <tbody> 
      <tr> 
       <td align="right">norm(t,d) &nbsp; = &nbsp; <code>doc.getBoost()</code> &nbsp;·&nbsp; lengthNorm &nbsp;·&nbsp;</td> 
       <td align="center" style="text-align:center">∏</td> 
       <td align="right"><code>f.getBoost</code>()</td> 
      </tr> 
      <tr> 
       <td>&nbsp;</td> 
       <td align="center" style="text-align:center"><small>field <em><strong>f</strong></em> in<em>d</em> named as <em><strong>t</strong></em></small></td> 
       <td>&nbsp;</td> 
      </tr> 
     </tbody> 
    </table>
    
      
      
    However the resulted *norm* value is `encoded` as a single byte before being stored. At search time, the norm byte value is read from the index`directory` and `decoded` back to a float *norm* value. This encoding/decoding, while reducing index size, comes with the price of precision loss - it is not guaranteed that*decode(encode(x)) = x*. For instance, *decode(encode(0.89)) = 0.75*. \[翻译\]然而norm值的结果在存储前将被编码成一个字节。在搜索时，norm字节从索引目录中读出，然后解码成一个浮点值。这个编码解码过程，虽然减少了空间使用，但是造成了精度损失。  
      
    Compression of norm values to a single byte saves memory at search time, because once a field is referenced at search time, its norms - for all documents - are maintained in memory. \[翻译\]norm值的压缩节省了空间，因为一旦一个field在搜索时被参考，那么所有文档的norm值将载入内存（虐心啊……）  
      
    The rationale supporting such lossy compression of norm values is that given the difficulty (and inaccuracy) of users to express their true information need by a query, only big differences matter. \[翻译\]支持压缩损失的理性解释是对于用户表达一个查询的真实信息的需求的难度来说，只有差距十分大时才会产生问题。  
      
    Last, note that search time is too late to modify this *norm* part of scoring, e.g. by using a different`Similarity` for search.  \[翻译\]最后，注意，搜索时间可能太长以至于来不及修改评分过程的Norm部分。例如，使用一个不同的Similarity

[Introduction To Information Retrieval_ Chapter 6]: http://nlp.stanford.edu/IR-book/html/htmledition/queries-as-vectors-1.html
[Boolean model _BM_ of Information Retrieval]: http://en.wikipedia.org/wiki/Standard_Boolean_model
[Vector Space Model _VSM_ of Information Retrieval]: http://en.wikipedia.org/wiki/Vector_Space_Model
[Tf-idf]: http://en.wikipedia.org/wiki/Tfidf
[Cosine Similarity]: http://en.wikipedia.org/wiki/Cosine_similarity
[dot product]: http://en.wikipedia.org/wiki/Dot_product
[Euclidean norms]: http://en.wikipedia.org/wiki/Euclidean_norm#Euclidean_norm