抖音基本信息爬虫

怼烎@ 2023-06-17 12:54 20阅读 0赞

版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。  
本文链接：https://blog.csdn.net/weixin\_43582101/article/details/92658860  
抓包。抓到了一个share\_url

![在这里插入图片描述][20191129092243223.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70]  
我访问过去看了下，  
https://www.iesdouyin.com/share/user/102064772608  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 1]

有数据的。那就拿数据。

然后发现它的数字做了字符集映射。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 2]  
搞他。

下载.ttf的文件，s3a.bytecdn.cn/ies/resource/falcon/douyin\_falcon/static/font/iconfont\_da2e2ef.ttf，这个便用font creator软件打开，看到这个图片我们就明白了字体与数字的关系  
![在这里插入图片描述][20191129092417406.png]

既然我们看到num\_对应数字1，num\_8对应数字7，那这个num\_8们怎么得到，与‘ ’有啥关系？

2.这个时候，需要大家安装pip install fontTools,使用fontTool打开ttf文件转化成xml文件，

采用下面代码

from fontTools.ttLib import TTFont  
font\_1 = TTFont(‘douyin.ttf’)  
font\_1.saveXML(‘font\_1.xml’)

我们看到了font\_1.xml，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 3]  
这就能看出来了。

import re  
def getDouyinNum(douIDNumCode):  
mapCode2Font = \{  
‘0xe602’: ‘num\_’,  
‘0xe603’: ‘num\_1’,  
‘0xe604’: ‘num\_2’,  
‘0xe605’: ‘num\_3’,  
‘0xe606’: ‘num\_4’,  
‘0xe607’: ‘num\_5’,  
‘0xe608’: ‘num\_6’,  
‘0xe609’: ‘num\_7’,  
‘0xe60a’: ‘num\_8’,  
‘0xe60b’: ‘num\_9’,  
‘0xe60c’: ‘num\_4’,  
‘0xe60d’: ‘num\_1’,  
‘0xe60e’: ‘num\_’,  
‘0xe60f’: ‘num\_5’,  
‘0xe610’: ‘num\_3’,  
‘0xe611’: ‘num\_2’,  
‘0xe612’: ‘num\_6’,  
‘0xe613’: ‘num\_8’,  
‘0xe614’: ‘num\_9’,  
‘0xe615’: ‘num\_7’,  
‘0xe616’: ‘num\_1’,  
‘0xe617’: ‘num\_3’,  
‘0xe618’: ‘num\_’,  
‘0xe619’: ‘num\_4’,  
‘0xe61a’: ‘num\_2’,  
‘0xe61b’: ‘num\_5’,  
‘0xe61c’: ‘num\_8’,  
‘0xe61d’: ‘num\_9’,  
‘0xe61e’: ‘num\_7’,  
‘0xe61f’: ‘num\_6’,  
\}  
mapFont2Num = \{  
‘num\_’: 1,  
‘num\_1’: 0,  
‘num\_2’: 3,  
‘num\_3’: 2,  
‘num\_4’: 4,  
‘num\_5’: 5,  
‘num\_6’: 6,  
‘num\_7’: 9,  
‘num\_8’: 7,  
‘num\_9’: 8,  
\}  
douIDNum = ‘’  
map1 = \{\}  
if douIDNumCode == ‘’:  
return ‘’  
for i in douIDNumCode:  
j = i.replace(’ &\#’, ‘0’).replace(’; ', ‘’)  
map1\[j\] = str(mapFont2Num\[mapCode2Font\[j\]\])  
return map1

def req(share\_id):  
url2= ‘https://www.iesdouyin.com/share/user/\{\}’.format(share\_id)  
headers =\{  
“User-Agent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_13\_6) "  
"AppleWebKit/537.36 (KHTML, like Gecko) "  
“Chrome/74.0.3729.169 Safari/537.36”  
\}  
res=requests.get(url2,verify=False,headers=headers)  
return res

def parse(res):  
description\_header = re.findall(‘抖音ID： (.*?)*

‘,res.text,re.S)  
description\_back = re.findall(’ *(.?)*’,res.text,re.S)  
mapDict = getDouyinNum(douIDNumCode=description\_back)  
res = description\_header\[0\]  
for i,j in mapDict.items():  
res = res.replace(’  *&\#’+i\[1:\]+’;* ',j)  
print(res)

if **name** == ‘**main**’:  
res = req(102064772608)  
parse(res)  
![在这里插入图片描述][20191129092639196.png]  
![在这里插入图片描述][20191129092704336.png]  
2019年8月更新（完整版）：  
检测到最近很多同学都在看这篇文章，我就重新写了一遍，思路还是跟上面一样，这次获取的是所有的用户信息。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 4]  
如果有用，还请点赞或者留言 ?

import re  
import requests  
from lxml import etree  
‘’’  
抖音用户基本信息 -> 请求share来获取数据  
‘’’  
def handle\_decode(input\_data):  
\# 匹配icon font

regex_list = [
        {'name': ['  ', '  ', '  '], 'value': 0},
        {'name': ['  ', '  ', '  '], 'value': 1},
        {'name': ['  ', '  ', '  '], 'value': 2},
        {'name': ['  ', '  ', '  '], 'value': 3},
        {'name': ['  ', '  ', '  '], 'value': 4},
        {'name': ['  ', '  ', '  '], 'value': 5},
        {'name': ['  ', '  ', '  '], 'value': 6},
        {'name': ['  ', '  ', '  '], 'value': 7},
        {'name': ['  ', '  ', '  '], 'value': 8},
        {'name': ['  ', '  ', '  '], 'value': 9},
    ]
    
    for i1 in regex_list:
        for i2 in i1['name']:
            input_data = re.sub(i2, str(i1['value']), input_data)       # 把正确value替换到自定义字体上
    html = etree.HTML(input_data)
    douyin_info = {}
    # 获取昵称
    douyin_info['nick_name'] = html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]
    # 获取抖音ID
    douyin_id = html.xpath("//div[@class='personal-card']/div[@class='info1']/p[@class='shortid']//text()")
    douyin_info['douyin_id'] = ''.join(douyin_id).replace('抖音ID：', '').replace(' ', '')
    
    # 职位类型
    try:
        douyin_info['job'] = html.xpath("//div[@class='personal-card']/div[@class='info2']/div[@class='verify-info']/span[@class='info']/text()")[0].strip()
    except:
        pass
    # 描述
    douyin_info['describe'] = html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='signature']/text()")[0].replace('\n', ',')
    # 关注
    douyin_info['follow_count'] = html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='focus block']//i[@class='icon iconfont follow-num']/text()")[0].strip()
    # 粉丝
    fans_value = ''.join(html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']//i[@class='icon iconfont follow-num']/text()"))
    unit = html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']/span[@class='num']/text()")
    if unit[-1].strip() == 'w':
        douyin_info['fans'] = str(float(fans_value) / 10) + 'w'
    else:
        douyin_info['fans'] = fans_value
    # 点赞
    like = ''.join(html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']//i[@class='icon iconfont follow-num']/text()"))
    unit = html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']/span[@class='num']/text()")
    if unit[-1].strip() == 'w':
        douyin_info['like'] = str(float(like) / 10) + 'w'
    else:
        douyin_info['like'] = like
    
    return douyin_info

def handle\_douyin\_info(url):  
header = \{  
‘user-agent’: ‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36’  
\}  
response = requests.get(url=url, headers=header)  
return handle\_decode(response.text)

if **name** == ‘**main**’:  
url = ‘https://www.iesdouyin.com/share/user/102064772608’  
print(handle\_douyin\_info(url))  
————————————————  
版权声明：本文为CSDN博主「考古学家lx」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。  
原文链接：https://blog.csdn.net/weixin\_43582101/article/details/92658860

[20191129092243223.png]: https://img-blog.csdnimg.cn/20191129092243223.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191129092308129.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191129092319487.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191129092340307.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx,size_16,color_FFFFFF,t_70
[20191129092417406.png]: https://img-blog.csdnimg.cn/20191129092417406.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2019112909251579.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx,size_16,color_FFFFFF,t_70
[20191129092639196.png]: https://img-blog.csdnimg.cn/20191129092639196.png
[20191129092704336.png]: https://img-blog.csdnimg.cn/20191129092704336.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191129092834890.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQyMzA2MDQx,size_16,color_FFFFFF,t_70