python3爬取斗鱼主播信息

男娘i 2023-06-23 10:54 23阅读 0赞

##           爬取舞蹈区的主播信息如昵称，热度，直播标题，直播地址，主播头像 ##

# #

# 1、抓包分析（[https://www.douyu.com/g\_dance][https_www.douyu.com_g_dance]）  #

[g\_dance][https_www.douyu.com_g_dance]可以换成其他的，原理是一样的

通过首页分析得不到要请求的链接，如果是直接通过elements中的元素来解析的话得不到主播的头像url地址，通过分析是通过json文件来传值的，第一页找不到，换到第二页仔细找的话发现![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70][]

这里返回的值经过对比是一致的，来到Headers这里拿url到浏览器中打开

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 1][]

[https://www.douyu.com/gapi/rkc/directory/2\_1008/2][https_www.douyu.com_gapi_rkc_directory_2_1008_2]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 3][]

可以去百度下个插件jsonview就可以看到这样了

![20191218164540989.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 4][]

对比信息完全一致

[https://www.douyu.com/gapi/rkc/directory/2\_1008/2][https_www.douyu.com_gapi_rkc_directory_2_1008_2] 这个链接最后的 2 换成 1 的话应该就是第一页了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 5][]

发现头像的url找不到，这时候回到详情页面中抓包头像的url

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 6][]

发现红色框的url最后那一部分是一样的，你们可以找多几个链接对比，结果还是一样的，把前面和后面拼接起来就是一个完整的url 了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 7][]

到此数据分析就结束了，就该动手写代码了

## 2、提取链接（[https://www.douyu.com/gapi/rkc/directory/2\_1008/][https_www.douyu.com_gapi_rkc_directory_2_1008_2]1） ##

# 3、数据处理 #

import requests
    import json
    import urllib.request
    import urllib
    
    
    class Douyu(object):
    
        def __init__(self):
            self.base_url = 'https://www.douyu.com/gapi/rkc/directory/2_1008/'
            self.header = {
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"
            }
            # 保存每个主播的数据
            self.datas = []
            # 拼接图片的地址
            self.url = 'https://www.douyu.com/'
    
        # 发请求
        def get_request(self, i):
            self.base_url = self.base_url + str(i)
            data = requests.get(self.base_url, headers=self.header).content.decode("utf-8")
            return data
    
        # 解析数据
        def parse_data(self, data):
            """
            data返回的数据 字符串格式 {"code":0,"msg":"success","data":{"ct":{"iv":0,"ivcv":0,"tag":1,"tn":"","vmcrr":0,"vmcm":""},"rl":[{"rid":554559,"rn":"7点左右直播。","uid":2826
            将字符串转为json格式json.loads  里面的data 下的 rl才是我们需要的，为data['data']['rl']
            （将json转为字符串格式json.dumps  【小知识】）
            """
            # 字符串转json ==》 dict 返回值
    
            data = json.loads(data)
    
            # 转换为字符串   json.dumps
            detail_lists = data['data']['rl']
            for detail in detail_lists:
                # 保存每个主播的信息
                dict_list = {}
                # 直播间地址
                dict_list['tar_url'] = self.url + str(detail['rid'])
                # 直播间标题
                dict_list['title'] = detail['rn']
                # 主播昵称
                dict_list['anchor'] = detail['nn']
                # 热度
                dict_list['hot'] = detail['ol']
                # 主播头像url
                dict_list['img_url'] = 'https://apic.douyucdn.cn/upload/' + detail['av'] + '_middle.jpg'
                # 每一个主播的信息都保存到一个list中
                self.datas.append(dict_list)
                print(dict_list)
                # 保存头像
                self.save_image(dict_list['img_url'],dict_list['anchor'])
    
        # 3、保存网页
        def save_data(self, data):
            #在get_request方法中删除decode("utf-8")    二进制保存网页
            #data = requests.get(self.base_url, headers=self.header).content
            with open("douyu.html", "wb") as f:
                f.write(data)
    
        # 保存json文件
        def save_json(self):
            json.dump(self.datas, open("anchor.json", 'w'))
    
        # 保存封面头像
        def save_image(self, url, name):
            try:
                request = urllib.request.Request(url)
                response = urllib.request.urlopen(request)
                get_img = response.read()
                # 需要事先创建好Img目录或者指定已存在的路径
                with open('Img/' + name + '.jpg', 'wb') as fp:
                    fp.write(get_img)
            except Exception as e:
                print("访问空")
    
        # 启动爬虫
        def run(self):
            # 页数  [1,3)
            for x in (1, 3):
                data = self.get_request(x)
                self.parse_data(data)
            self.save_json()
    
    Douyu().run()

爬取结束，想要存入数据库的话自己加个方法入库吧这里就不多写了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 9][]

会有json文件生成

![20191218171552584.png][]

到   [http://www.bejson.com/][http_www.bejson.com]  格式化json文件比较好看

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 10][]

[https_www.douyu.com_g_dance]: https://www.douyu.com/g_dance
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191218164151792.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191218164310377.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[https_www.douyu.com_gapi_rkc_directory_2_1008_2]: https://www.douyu.com/gapi/rkc/directory/2_1008/2
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191218164640888.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191218164638442.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[20191218164540989.png]: https://img-blog.csdnimg.cn/20191218164540989.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191218164729287.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191218164819364.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20191218165228976.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191218165150756.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20191218171421695.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20191218171439377.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70
[20191218171552584.png]: https://img-blog.csdnimg.cn/20191218171552584.png
[http_www.bejson.com]: http://www.bejson.com/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20191218171805516.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpOTM2NzU=,size_16,color_FFFFFF,t_70