文本查重（未完）

心已赠人 2022-03-29 03:16 238阅读 0赞

因为申请的学校的项目，准备写个文本查重。

浏览了些网上的资料，结合下自己的实际需求，觉得simhash算法比较合适一些。

先附上一些参考资料吧

需求总结不错

[https://blog.csdn.net/weixin\_43098787/article/details/82836140][https_blog.csdn.net_weixin_43098787_article_details_82836140]

simhash算法入门了解

[https://blog.csdn.net/chinafire525/article/details/78645956][https_blog.csdn.net_chinafire525_article_details_78645956]

simhash更全面的介绍（引用文献不错）

[https://blog.csdn.net/sunny\_ss12/article/details/46958155][https_blog.csdn.net_sunny_ss12_article_details_46958155]

下面就是一些实现了：

[https://blog.csdn.net/weixin\_43098787/article/details/82838348][https_blog.csdn.net_weixin_43098787_article_details_82838348]

py3：[https://blog.csdn.net/gzt940726/article/details/80460419][https_blog.csdn.net_gzt940726_article_details_80460419]

[https://www.cnblogs.com/Liqiongyu/p/6213323.html][https_www.cnblogs.com_Liqiongyu_p_6213323.html]

高效计算二进制序列中1的个数：

[https://yanyiwu.com/work/2014/01/30/simhash-shi-xian-xiang-jie.html][https_yanyiwu.com_work_2014_01_30_simhash-shi-xian-xiang-jie.html]

simhash海量文本去重工程化

[https://yuerblog.cc/2018/05/30/simhash-text-unique-arch/][https_yuerblog.cc_2018_05_30_simhash-text-unique-arch]

汉明距离

[http://www.lanceyan.com/tag/simhash][http_www.lanceyan.com_tag_simhash]

关键词提取

[https://blog.csdn.net/gzt940726/article/details/80256011][https_blog.csdn.net_gzt940726_article_details_80256011]

[https_blog.csdn.net_weixin_43098787_article_details_82836140]: https://blog.csdn.net/weixin_43098787/article/details/82836140
[https_blog.csdn.net_chinafire525_article_details_78645956]: https://blog.csdn.net/chinafire525/article/details/78645956
[https_blog.csdn.net_sunny_ss12_article_details_46958155]: https://blog.csdn.net/sunny_ss12/article/details/46958155
[https_blog.csdn.net_weixin_43098787_article_details_82838348]: https://blog.csdn.net/weixin_43098787/article/details/82838348
[https_blog.csdn.net_gzt940726_article_details_80460419]: https://blog.csdn.net/gzt940726/article/details/80460419
[https_www.cnblogs.com_Liqiongyu_p_6213323.html]: https://www.cnblogs.com/Liqiongyu/p/6213323.html
[https_yanyiwu.com_work_2014_01_30_simhash-shi-xian-xiang-jie.html]: https://yanyiwu.com/work/2014/01/30/simhash-shi-xian-xiang-jie.html
[https_yuerblog.cc_2018_05_30_simhash-text-unique-arch]: https://yuerblog.cc/2018/05/30/simhash-text-unique-arch/
[http_www.lanceyan.com_tag_simhash]: http://www.lanceyan.com/tag/simhash
[https_blog.csdn.net_gzt940726_article_details_80256011]: https://blog.csdn.net/gzt940726/article/details/80256011