Python爬取比比网中标标书并保存成PDF格式

雨点打透心脏的1/2处 2022-12-19 03:58 134阅读 0赞

## 前言 ##

本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理。

PS：如有需要Python学习资料的小伙伴可以加点击下方链接自行获取

[python免费学习资料以及群交流解答点击即可加入][python]

--------------------

**python开发环境**

*  python 3.6
 *  pycharm

import requests
    import parsel
    import pdfkit
    import time

相关模块pip安装即可

## 目标网页分析 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center]  
1、先从列表页中获取详情页的URL地址

是静态网站，可以直接请求网页获取数据  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 1]

for page in range(1, 31):
        url = 'https://www.bibenet.com/mfzbu{}.html'.format(page)
        headers = { 
    
            'Referer': 'https://www.bibenet.com/mianfei/',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
        }
        response = requests.get(url=url, headers=headers)
        selector = parsel.Selector(response.text)
        urls = selector.css('body > div.wrap > div.clearFiex > div.col9.fl > div.secondary_box > table tr .fl a::attr(href)').getall()
        for page_url in urls:
        print(page_url)

2、从详情页中获取标题以及内容

response_2 = requests.get(url=page_url, headers=headers)
    selector_2 = parsel.Selector(response_2.text)
    article = selector_2.css('.container').get()
    title = selector_2.css('.detailtitle::text').get()

3、保存html网页数据并转成PDF

html_str = """ <!doctype html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Document</title> </head> <body> {article} </body> </html> """
    def download(article, title):
        html = html_str.format(article=article)
        html_path = 'D:\\python\\demo\\招标网\\文书\\' + title + '.html'
        pdf_path = 'D:\\python\\demo\\招标网\\文书\\' + title + '.pdf'
        with open(html_path, mode='wb', encoding='utf-8') as f:
            f.write(html)
            print('{}已下载完成'.format(title))
        # exe 文件存放的路径
        config = pdfkit.configuration(wkhtmltopdf='C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltopdf.exe')
        # 把 html 通过 pdfkit 变成 pdf 文件
        pdfkit.from_file(html_path, pdf_path, configuration=config)

## 运行实现效果 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 3]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 4]

[python]: https://jq.qq.com/?_wv=1027&k=ThcV4SNF
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/cd009fcdc2e84756be39b6a9791a832e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/fd73ebd350684c5c911f1028f441109d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/2020110514373812.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/20201105143749920.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA_size_16_color_FFFFFF_t_70_pic_center 4]: https://img-blog.csdnimg.cn/20201105143827368.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaTM0Nzc5NTc5MA==,size_16,color_FFFFFF,t_70#pic_center