爬虫爬取在线小说阅读网站详解

小鱼儿 2023-10-18 10:55 61阅读 0赞

### 前言 ###

环境：python 安装、requests安装、Beautiful Soup安装

爬取目标：

笔趣看网站的《校花之贴身高手》，以下是第一章链接

[https://www.biqukan.com/11\_11499/4260511.html][https_www.biqukan.com_11_11499_4260511.html]

### 开始爬取 ###

1.打开链接，打开检查窗口

通过审查Elements,能定位到小说的文本内容在<div id="content" class="showtxt">里面

![Image 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70][]  
2.借用requests库将页面的HTML内容先爬取下来

新建文件getnovel1.py

import requests
    
    if __name__ == '__main__':
        print((requests.get(url='https://www.biqukan.com/11_11499/4260511.html')).text)

然后运行代码：

![Image 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 1][]

### 解析HTML信息 ###

爬取网页的HTML信息后，就要使用Beautiful Soup 来解析HTML信息

1.安装Beautiful Soup

pip install beautifulsoup4

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 2][]

2.在解析html之前，我们需要创建一个Beautiful Soup 对象。

Beautiful Soup函数的参数就是之前获取到的html信息。

然后使用find\_all()方法,获得html信息中所有class属性为showtxt的div标签。

find\_all方法的第一个参数是获取的标签名，第二个参数是标签的属性，带了一个下划线是因为class是python中的关键字。

新建文件 getnovel2.py

import requests
    from bs4 import BeautifulSoup
    
    if __name__=="__main__":
        html = requests.get(url='https://www.biqukan.com/11_11499/4260511.html').text
        novel = BeautifulSoup(html).find_all('div',class_='showtxt')
        print(novel)

运行：

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 3][]

3.但是爬取的内容还是有一些不如人意，我们要想法去除掉div 标签、br标签等。

新建文件getnovel3.py

分析：

find\_all匹配的返回结果是列表，使用tnovel\[0\]获取匹配结果，使用text属性，提取文本内容，  
这样能过滤掉br标签。

然后使用replace方法，替换空格为回车，为什么是8格空格，再看网页结构。

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 4][]

运行结果：

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 5][]

### 爬取目录信息 ###

1.上面已经爬取了第一章的内容，想要下载所有章节，先来分析下小说目录。

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 6][]

2.从上面我们知道，这些章节存放在了class属性为listmain的div标签下的dl下的dd下的a标签中，

而且这些a标签的href属性前面都是一样的，只有后面部分不一样。

3.对于BeautifulSoup返回的匹配结果a，使用a.get('href')方法就能获取href的属性值，使用a.string就能获取章节的名字。

4.新建文件getnovel4.py

from bs4 import BeautifulSoup
    import requests
    if __name__ == "__main__":
         server = 'https://www.biqukan.com/'
         target = 'https://www.biqukan.com/11_11499/'
         req = requests.get(url = target)
         html = req.text
         div_bf = BeautifulSoup(html)
         div = div_bf.find_all('div', class_ = 'listmain')
         a_bf = BeautifulSoup(str(div[0]))
         a = a_bf.find_all('a')
         for each in a:
              print(each.string, server + each.get('href'))

备注：

小说每张章的链接放在了class属性为listmain的div标签下的<a>中。先匹配class属性为listmain的<div>标签，再匹配

<a>标签。接下来再匹配每一个<a>标签，并提取章节名和章节文章。因为find\_all返回的是一个列表，里边存放了很多的<a>标签，

所以使用for 循环遍历每个<a>标签并打印出来。

运行：

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 7][]

5.整合代码

# -*- coding:UTF-8 -*-
    from bs4 import BeautifulSoup
    import requests, sys
    
    
    class downloader(object):
    
        def __init__(self):
            self.server = 'http://www.biqukan.com/'
            self.target = 'https://www.biqukan.com/11_11499/'
            self.names = []            #存放章节名
            self.urls = []            #存放章节链接
            self.nums = 0            #章节数
    
    
        def get_download_url(self):
            req = requests.get(url = self.target)
            html = req.text
            div_bf = BeautifulSoup(html)
            div = div_bf.find_all('div', class_ = 'listmain')
            a_bf = BeautifulSoup(str(div[0]))
            a = a_bf.find_all('a')
            self.nums = len(a[15:])                                #剔除不必要的章节，并统计章节数
            for each in a[15:]:
                self.names.append(each.string)
                self.urls.append(self.server + each.get('href'))
    
    
        def get_contents(self, target):
            req = requests.get(url = target)
            html = req.text
            bf = BeautifulSoup(html)
            texts = bf.find_all('div', class_ = 'showtxt')
            texts = texts[0].text.replace('\xa0'*8,'\n\n')
            return texts
    
    
        def writer(self, name, path, text):
            write_flag = True
            with open(path, 'a', encoding='utf-8') as f:
                f.write(name + '\n')
                f.writelines(text)
                f.write('\n\n')
    
    if __name__ == "__main__":
        dl = downloader()
        dl.get_download_url()
        print('《小说》开始下载：')
        for i in range(dl.nums):
            dl.writer(dl.names[i], '小说.txt', dl.get_contents(dl.urls[i]))
            sys.stdout.write("  已下载:%.3f%%" %  float(i/dl.nums) + '\r')
            sys.stdout.flush()
        print('《小说》下载完成')

### 运行 ###

![Image 1][]

![Image 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 9][]

[https_www.biqukan.com_11_11499_4260511.html]: https://www.biqukan.com/11_11499/4260511.html
[Image 1]: 
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20181106223758396.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20181106223813654.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20181106223825928.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20181106223842494.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20181106223902435.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20181106223908294.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20181106223913449.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20181106223925630.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20181106223939932.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20181106223945624.PNG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk=,size_16,color_FFFFFF,t_70