NLP-文本处理：词的过滤（Words Filtering）【停用词、低频词】

柔情只为你懂 2023-10-04 09:32 15阅读 0赞

类似于特征筛选的过程，如果不过滤掉它们，很可能对我们的语言模型产生负面的影响。

*  停用词，对于理解文章没有太大意义的词，比如"the"、“an”、“his”、“their”，但是，也需要考虑自己的应用场景，来决定过滤哪些停用词。停用词下载：链接：https://pan.baidu.com/s/1XRNPAAQIkaNO-8qb\_DdctQ 提取码：86o0
 *  出现频率很低的词对分析作用不大，所以一般也会去掉。比如1万个单词的文章里它只出现了一两次。

把停用词、低频词过滤之后，即可得到一个词典库。

### 1、NLTK停用词库 ###

import nltk
    from nltk.corpus import stopwords
    stopset = set(stopwords.words('english'))

{
        'here', 'they', 'over', "you'd", 'at', 'more', "doesn't", 'again', 'isn', 'once', 'your', 'their', 'will', 'having', 'nor', 'them', "that'll", 'the', "should've", 'wouldn', 'ourselves', 'this', 'hers', 'were', 'me', 'own', 'up', 'all', "mustn't", 'under', 'do', 'our', "you're", 'did', 'most', 'myself', 'those', 'why', 'with', 'who', 'each', 'o', "didn't", "wasn't", 'on', 'about', 'm', 'of', 'just', 'while', 'can', 'themselves', 'y', 'as', 'if', 'very', 'in', 'for', 'above', 'you', 'd', 'further', 'was', 'its', 'through', 'between', 'being', 'has', 'wasn', 'to', 'which', 'few', 'down', "needn't", 'it', 'haven', 'his', 'some', 'hadn', "you'll", 'from', 'should', 'herself', 'so', 's', "aren't", 'only', 'didn', "wouldn't", 'yourself', 'before', 'been', 'have', 'ain', 'does', 'theirs', 'he', 'out', 'aren', 'an', 'these', "won't", 'ma', 'we', 'am', 'my', 'against', 'into', 're', "weren't", 'until', 'her', 'there', 'itself', 'a', 'where', 'now', "couldn't", 'needn', 'and', 'other', 'yourselves', 'won', 'himself', "isn't", 'after', 'is', "it's", 'what', 'when', "haven't", 'doing', 've', 'not', "she's", "don't", 'hasn', 'shan', 'be', 'but', 'shouldn', 'him', 'off', 'doesn', 'don', 'weren', 'during', 'how', 'too', 'she', 't', "you've", 'whom', 'both', 'couldn', 'same', 'then', 'mightn', 'had', 'than', 'yours', 'such', "mightn't", "shouldn't", 'i', 'by', 'any', 'no', "shan't", 'that', 'below', 'or', 'll', "hasn't", 'are', 'ours', "hadn't", 'mustn', 'because'}