nltk 处理文本

墨蓝 2023-07-03 12:24 22阅读 0赞

# ![20191009191333910.png][] #

# [日萌社][Link 1] #

[人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新）][AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]

--------------------

[nltk 处理文本][nltk]

[注意力机制 SENet、CBAM][SENet_CBAM]

[卷积神经网络 处理文本：word2vec、TF-IDF、TextRank、字符卷积、词卷积、卷积神经网络文本分类模型的实现(Conv1D一维卷积、Conv2D二维卷积)][_word2vec_TF-IDF_TextRank_Conv1D_Conv2D]

[反向传播、链式求导][Link 2]

[梯度下降][Link 3]

[最小二乘法(LS算法)：实际为L2范数的一个具体应用(计算残差平方和)][LS_L2]

[线性回归 例子][Link 4]

--------------------

1.安装
    	conda install nltk
    	>>> import nltk
    	>>> nltk.download() #该步骤打开下载列表可能需要翻墙才能连接远程服务器
    	showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70][]

2.nltk 安装停用词模块

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1][]

测试nltk的停用词模块
    		>>> from nltk.corpus import stopwords
    		>>> stoplist = stopwords.words('english')
    		>>> stoplist
    		['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 
    		 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 
    		 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 
    		 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 
    		 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 
    		 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 
    		 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 
    		 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 
    		 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 
    		 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', 
    		 "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 
    		 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]
    
    3.还原词干，可把“ing/ed”等后缀丢弃，“ies”用“y”代替，这个词的所有变形都还原为只有一个词干，即对这部分词根的处理
    	>>> from nltk import *
    	>>> PorterStemmer().stem("going")
    	'go'

[20191009191333910.png]: https://img-blog.csdnimg.cn/20191009191333910.png
[Link 1]: http://www.rimengshe.com/
[AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]: https://blog.csdn.net/zimiao552147572/article/details/88867161
[nltk]: https://azusa.blog.csdn.net/article/details/104203185
[SENet_CBAM]: https://azusa.blog.csdn.net/article/details/104201845
[_word2vec_TF-IDF_TextRank_Conv1D_Conv2D]: https://azusa.blog.csdn.net/article/details/104208463
[Link 2]: https://azusa.blog.csdn.net/article/details/104201797
[Link 3]: https://azusa.blog.csdn.net/article/details/104201755
[LS_L2]: https://azusa.blog.csdn.net/article/details/104201735
[Link 4]: https://azusa.blog.csdn.net/article/details/104201717
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200206215909120.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020020621591345.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70