中文分词之隐马尔可夫模型

Bertha 。 2022-03-15 03:04 280阅读 0赞

隐马尔可夫模型理论知识：  
角色：收拾烂摊子的角色  
1) 生成方式  
2) 路径选择（viterbi算法）——>动态规划

上篇文章讲到语言模型（1,2,3）  
二元语言模型 == 一阶马尔科夫模型

马尔科夫模型有3类重要参数：  
1、状态  
2、初始概率

初始概率计算方法举例：  
    假设有100篇文章，  
    时光荏苒 30篇   -> 30 / 100  
    今天 10篇  
    开心 5篇  
    加班 1篇

3.转移概率计算方法举例：

x x x x x x x x x x x x x x A B x x x x x A B x x x x  A C x x x x  
P(B|A) = 2/3  
P(C|A) = 1/3

计算的前提条件：大量的文章

因为马尔科夫模型是针对一个序列建模的，现实中会有多个序列（比如语音识别），因此而产生隐马尔可夫模型

双序列举例：

词性标注![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70][]

隐马尔科夫模型中有5类重要参数：  
1、状态，M个 = 4（位置信息）\*20（词性信息） = 80（大概有80个状态）  
2、初始概率：M个  
3、转移概率：M\*M（一个状态到另外一个状态的概率）  
4、发射概率：M\*N  
5、观察，N个 = 2000（有多少个汉字，就有多少个观察）

BMES状态：举例：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 1][]

初始概率：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 2][]

获取方法：统计

假设有100篇文章，  
    时光荏苒 30篇   那么初始概率为-> 30 / 100

转移概率：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 3][]

状态概率：每一个状态到一个汉字的概率

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 5][]

P(S1,S2,S3,O1,O2,O3) 联合概率  
= P(S1) \* P(O1|S1) \* P(S2|S1) \*\*\*  \*\*

![20190228222746608.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 6][]

viterbi动态规划最优路径算法

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 7][]

目的：将零散的词汇粘在一起

实践：倒排索引

![20190301090012567.png][]

1.原始数据：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 8][]

思路：

1.map做分词：name->token,token,token

2.reduce(聚合)  
token1 -> name name name

代码：

### map.py: ###

\#!/usr/bin/python

import os  
import sys

os.system('tar xvzf jieba.tar.gz > /dev/null')

reload(sys)  
sys.setdefaultencoding('utf-8')

sys.path.append("./")

import jieba  
import jieba.posseg  
import jieba.analyse

def mapper\_func():  
    for line in sys.stdin:  
        ss = line.strip().split('\\t')  
        if len(ss) != 2:  
            continue  
        music\_id = ss\[0\].strip()  
        music\_name = ss\[1\].strip()

tmp\_list = \[\]  
        for x, w in jieba.analyse.extract\_tags(music\_name, withWeight=True):  
            tmp\_list.append((x, float(w)))

final\_token\_score\_list = sorted(tmp\_list, key=lambda x: x\[1\], reverse=True)

print '\\t'.join(\[music\_id, music\_name, ''.join(\[''.join(\[t\_w\[0\], str(t\_w\[1\])\]) for t\_w in final\_token\_score\_list\])\])

if \_\_name\_\_ == "\_\_main\_\_":  
    module = sys.modules\[\_\_name\_\_\]  
    func = getattr(module, sys.argv\[1\])  
    args = None  
    if len(sys.argv) > 1:  
        args = sys.argv\[2:\]  
    func(\*args)

###    倒排map\_inverted.py： ###

\#!/usr/bin/python

import os  
import sys

def mapper\_func():  
    for line in sys.stdin:  
        ss = line.strip().split('\\t')  
        if len(ss) != 3:  
            continue  
        music\_id = ss\[0\].strip()  
        music\_name = ss\[1\].strip()  
        music\_fealist = ss\[2\].strip()

for fea in music\_fealist.split(''):  
            token, weight = fea.strip().split('')  
            print '\\t'.join(\[token, music\_name, weight\])

### red\_inverted.py: ###

\#!/usr/bin/python

import os  
import sys

def reducer\_func():

cur\_token = None  
    m\_list = \[\]

for line in sys.stdin:  
        ss = line.strip().split('\\t')  
        if len(ss) != 3:  
            continue  
        token = ss\[0\].strip()  
        name = ss\[1\].strip()  
        weight = float(ss\[2\].strip())

if cur\_token == None:  
            cur\_token = token  
        if cur\_token != token:

final\_list = sorted(m\_list, key=lambda x: x\[1\], reverse=True)  
            print '\\t'.join(\[cur\_token, ''.join(\[''.join(\[name\_weight\[0\], str(name\_weight\[1\])\]) for name\_weight in final\_list\])\])

cur\_token = token  
            m\_list = \[\]

m\_list.append((name, weight))

final\_list = sorted(m\_list, key=lambda x: x\[1\], reverse=True)  
    print '\\t'.join(\[cur\_token, ''.join(\[''.join(\[name\_weight\[0\], str(name\_weight\[1\])\]) for name\_weight in final\_list\])\])

脚本run.sh：

HADOOP\_CMD="/usr/local/src/hadoop-2.6.5/bin/hadoop"  
STREAM\_JAR\_PATH="/usr/local/src/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar  
"

INPUT\_FILE\_PATH\_1="/music\_meta.txt.small"  
OUTPUT\_Z\_PATH="/output\_z\_fenci"  
OUTPUT\_D\_PATH="/output\_d\_fenci"

$HADOOP\_CMD fs -rmr $OUTPUT\_Z\_PATH  
$HADOOP\_CMD fs -rmr $OUTPUT\_D\_PATH

\# Step 1.  
$HADOOP\_CMD jar $STREAM\_JAR\_PATH \\  
    -input $INPUT\_FILE\_PATH\_1 \\  
    -output $OUTPUT\_Z\_PATH \\  
    -mapper "python map.py mapper\_func" \\  
    -jobconf "mapred.reduce.tasks=0" \\  
    -jobconf "mapreduce.map.memory.mb=4096" \\  
    -jobconf  "mapred.job.name=jieba\_fenci\_demo" \\  
    -file "./jieba.tar.gz" \\  
    -file "./map.py"

\# Step 2.  
$HADOOP\_CMD jar $STREAM\_JAR\_PATH \\  
    -input $OUTPUT\_Z\_PATH \\  
    -output $OUTPUT\_D\_PATH \\  
    -mapper "python map\_inverted.py mapper\_func" \\  
    -reducer "python red\_inverted.py reducer\_func" \\  
    -jobconf "mapred.reduce.tasks=2" \\  
    -jobconf "mapreduce.map.memory.mb=4096" \\  
    -jobconf  "mapred.job.name=jieba\_fenci" \\  
    -file "./map\_inverted.py" \\  
    -file "./red\_inverted.py"

第一阶段中间结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 9][]

最后结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 10][]

这就是召回：token检索item

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70]: /images/20220315/4a62f7c044f3425fabe0e5839091fd37.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 1]: /images/20220315/47a2b91d701046efb3991098eddc2344.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 2]: /images/20220315/3eb51c59e5d44f8dad3324e45a652bb0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 3]: /images/20220315/d51956e335b84410b6e1764b3da02ca2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 4]: /images/20220315/6bf35def104d4f19b4d19fd3536bc290.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 5]: /images/20220315/63100e2a6cb440f0baac24f6ab102a49.png
[20190228222746608.png]: /images/20220315/d7b367ba58f44635b701198782bb800d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 6]: /images/20220315/b797c440d77040e8bca6c47cf7027a40.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 7]: /images/20220315/aa7e6b61926e42c2bb27f8d669726154.png
[20190301090012567.png]: /images/20220315/29ec2068e7c342218538c12dd6d6461e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 8]: /images/20220315/76d0d84ee5354087acef18414cbddebb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 9]: /images/20220315/19f928f424934a7cb4945509d70fb3ac.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI4Mjg2MDI3_size_16_color_FFFFFF_t_70 10]: /images/20220315/87757bff81204518b35560d25b65e6c5.png