爬取三国演义全本内容，保存到TXT文件夹下（requests+bs4）

╰+哭是因爲堅強的太久メ 2022-10-22 07:59 168阅读 0赞

# 一、爬取界面分析 #

爬取网站：[https://www.shicimingju.com/book/sanguoyanyi.html][https_www.shicimingju.com_book_sanguoyanyi.html]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70]  
通过分析，该界面采用静态加载的方式呈现数据，即获取网页源代码可获取相应的数据，在本界面中获取的数据有章节的标题，以及章节内容的链接。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 1]  
在章节内容界面，同样时采用静态加载数据的方式。

# 二、程序源代码 #

import requests
    from bs4 import BeautifulSoup
    
    fp = open('./sanguo.txt','w',encoding='utf-8')
    headers = {
        
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
        }
    main_url = "https://www.shicimingju.com/book/sanguoyanyi.html"
    page_text = requests.get(url = main_url,headers = headers)
    page_text.encoding = page_text.apparent_encoding
    page_text = page_text.text
    soup = BeautifulSoup(page_text,'lxml')
    a_list = soup.select('.book-mulu > ul > li > a')
    for a in a_list:
        title = a.string
        detail_url = 'https://www.shicimingju.com' + a['href']
        page_text_detail = requests.get(detail_url,headers = headers)
        page_text_detail.encoding = page_text_detail.apparent_encoding
        page_text_detail = page_text_detail.text
        soup = BeautifulSoup(page_text_detail,'lxml')
        div_tag = soup.find('div',class_ = 'chapter_content')
        content = div_tag.text
        fp.write(title + ':' + content + '\n')
        print(title,'保存成功!!!')
    fp.close()

# 三、程序运行结果 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 2]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 3]

[https_www.shicimingju.com_book_sanguoyanyi.html]: https://www.shicimingju.com/book/sanguoyanyi.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70]: /images/20221022/e618642441654119bf054f51667ca29b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 1]: /images/20221022/90aab94ac75c44958fd1d957f4e0e24e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 2]: /images/20221022/2bb4f42df71d4ee2a81f646e72300e01.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dldHNfcw_size_16_color_FFFFFF_t_70 3]: /images/20221022/9141c37a2c1f40109c104155eb1627f0.png