爬虫案例一：基础

r囧r小猫 2023-08-17 17:19 82阅读 0赞

# 前言 #

我在秋招期间，有幸做了一家电商公司的线上题目，然后发现这个题目比较全面，所以我就准备以这个题目为核心写个系列教程，不过为了公平起见，我就不写公司名字了。

# 题目网站 #

*  网址：[http://glidedsky.com/login][http_glidedsky.com_login]
 *  需要注册一个账号，需要**邮箱验证**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70]
 *  注册成功后，就会看到第一个题目了，点击进入题目  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 1]
 *  题目描述和提交答案的地方  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 2]
 *  题目提供的网页  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 3]

# 如何分析 #

*  首先根据题目要求，我们得知我们的目标是**对一个页面上面的所有数字进行求和**
 *  所以我们先要解决如何获取页面内容，然后再解决如何解析获取到所有的数字标签

# 获取网页内容 #

*  这里我们使用`requests`库来作为我们的请求库
 *  打开浏览器**开发者工具**，点击**重新载入**，然后点击第一条请求，点击响应，然后我们发现页面内容正好是页面上所展示的，所以目标明确了，就是**模拟这条请求获取网页内容**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 4]
 *  点击消息头回到初始位置，这里就是请求的相关信息，具体参数说明请参考：[HTTP基本原理][HTTP]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 5]
 *  然后我们把**url、请求头**都复制出来，并且对请求头简单**格式处理**一下  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 6]
 *  然后就可以编辑代码了
    
        import requests
        url='http://glidedsky.com/level/web/crawler-basic-1'
        headers={ 
            'Host':'glidedsky.com',
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Language':'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
            'Accept-Encoding':'gzip, deflate',
            'Referer':'http://glidedsky.com/level/crawler-basic-1',
            'Connection':'keep-alive',
            'Cookie':'XSRF-TOKEN=eyJpdiI6ImNrWHNhYk5aQzNrdXBiQ0pSY2dNakE9PSIsInZhbHVlIjoiZHRzeGkzZ1wvaUpoOXM2VVVralpzbFB4bXh6OVdIWnNad2txXC9jWFB0ZFA3dDlUY2Vvcis2RnJ3WlFqRUhqOXI3IiwibWFjIjoiNDI1NWYxZGVhMjUxNTZmOTNmZDQ1MmQ4YzYzMDIwYTllNTc5YzI2ZmIyMTMzN2IyZGRlMmRmMGIwYTVkZmU3ZiJ9; glidedsky_session=eyJpdiI6InNJZ1J4WGplOUNXbFlqU29ndzc4WFE9PSIsInZhbHVlIjoiSVdhY3ZQOCt1Z0psVWNDR0dLSzAyaVg3NHdBUndWZTJmUExmekpTOEUrNDhqXC9GVWFvVVhCSEwzR1REbm54WWsiLCJtYWMiOiJhZTQ2OGE1OTVmYTlkODViZGI0NDEyNDNiZmJmOWI3YzE3Y2I3Y2Q5OGY5M2ZlMjY5Y2U4NDU3MWIzODliOGMxIn0%3D; footprints=eyJpdiI6ImpuR2xMRVRualwvS0lhS2NtTTFLOWJBPT0iLCJ2YWx1ZSI6IlwvUEYrWWJnRWRCQ09mS2ZXY2EwZ0FodFFXdUNsTkVSZHRmUWxoZ0xIQXBLSHQwMGpsenZISFVXRkxnandydkZLIiwibWFjIjoiZWRhMzU2NTU0ZmRlNzMxM2ZhOWVmN2VlMzBhODNiNGM2MmMzMWMxYzIyZGFhMjQzMzcyODVmZjFmYTdiZDc4ZCJ9; Hm_lvt_020fbaad6104bcddd1db12d6b78812f6=1573543327; Hm_lpvt_020fbaad6104bcddd1db12d6b78812f6=1573548587; _ga=GA1.2.1195179388.1573543328; _gid=GA1.2.165778833.1573543328',
            'Upgrade-Insecure-Requests':'1',
            'Pragma':'no-cache',
            'Cache-Control':'no-cache'
        }
        response=requests.get(url,headers=headers)
        print(response.text)
 *  运行结果，大概看了一下是对的（当然实际分析中最好是能将页面下载下来打开对比原网页）  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 7]
 *  **到此完成第一步，网页请求**

# 解析网页获取数字的值 #

*  拿到了网页数据后，我们还要通过**解析**获取到具体的数字而不是一堆**冗余的标签**
 *  流行的解析方法有很多种，`re正则、BeautifulSoup、xpath、pyquery`等，这里我用的是**BeautifulSoup**
 *  老样子，打开开发者工具，点击查看器，点击左边的**小鼠标按钮**，然后随便点击页面上的某个数字，下面就显示了你点击位置具体的标签了  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 8]
 *  然后点开这个`div`标签，发现了我们想要的数字  
    ![在这里插入图片描述][2019111217160721.png]
 *  然后观察这个标签的特征，它除了是`div`标签以外，还有个**class**属性为`col-md-1`
 *  所以我们解析的时候能依据的规则就是这两点，但是`div`标签不是很适合作为判断依据，因为其他地方也肯定有这个标签，所以**class**属性更适合作为**判断依据**
 *  然后利用`BeautifulSoup`的方法开始编辑代码
    
        import requests
        from bs4 import BeautifulSoup
        url='http://glidedsky.com/level/web/crawler-basic-1'
        headers={ 
            'Host':'glidedsky.com',
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Language':'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
            'Accept-Encoding':'gzip, deflate',
            'Referer':'http://glidedsky.com/level/crawler-basic-1',
            'Connection':'keep-alive',
            'Cookie':'XSRF-TOKEN=eyJpdiI6ImNrWHNhYk5aQzNrdXBiQ0pSY2dNakE9PSIsInZhbHVlIjoiZHRzeGkzZ1wvaUpoOXM2VVVralpzbFB4bXh6OVdIWnNad2txXC9jWFB0ZFA3dDlUY2Vvcis2RnJ3WlFqRUhqOXI3IiwibWFjIjoiNDI1NWYxZGVhMjUxNTZmOTNmZDQ1MmQ4YzYzMDIwYTllNTc5YzI2ZmIyMTMzN2IyZGRlMmRmMGIwYTVkZmU3ZiJ9; glidedsky_session=eyJpdiI6InNJZ1J4WGplOUNXbFlqU29ndzc4WFE9PSIsInZhbHVlIjoiSVdhY3ZQOCt1Z0psVWNDR0dLSzAyaVg3NHdBUndWZTJmUExmekpTOEUrNDhqXC9GVWFvVVhCSEwzR1REbm54WWsiLCJtYWMiOiJhZTQ2OGE1OTVmYTlkODViZGI0NDEyNDNiZmJmOWI3YzE3Y2I3Y2Q5OGY5M2ZlMjY5Y2U4NDU3MWIzODliOGMxIn0%3D; footprints=eyJpdiI6ImpuR2xMRVRualwvS0lhS2NtTTFLOWJBPT0iLCJ2YWx1ZSI6IlwvUEYrWWJnRWRCQ09mS2ZXY2EwZ0FodFFXdUNsTkVSZHRmUWxoZ0xIQXBLSHQwMGpsenZISFVXRkxnandydkZLIiwibWFjIjoiZWRhMzU2NTU0ZmRlNzMxM2ZhOWVmN2VlMzBhODNiNGM2MmMzMWMxYzIyZGFhMjQzMzcyODVmZjFmYTdiZDc4ZCJ9; Hm_lvt_020fbaad6104bcddd1db12d6b78812f6=1573543327; Hm_lpvt_020fbaad6104bcddd1db12d6b78812f6=1573548587; _ga=GA1.2.1195179388.1573543328; _gid=GA1.2.165778833.1573543328',
            'Upgrade-Insecure-Requests':'1',
            'Pragma':'no-cache',
            'Cache-Control':'no-cache'
        }
        response=requests.get(url,headers=headers)
        #生成解析器对象
        soup=BeautifulSoup(response.text,'lxml')
        #根据标签的特征获取所有满足特征的标签
        divs=soup.find_all('div',attrs={ 'class':'col-md-1'})
        #循环每个标签对象 对象的text属性是标签的值 strip()是去掉字符串两边的空格
        for div in divs:
            print(div.text.strip())
 *  输出结果就是我们想要的数字了  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 9]

# 最终代码 #

*  计算最终的总和就是把获取到的值**转化一下数据类型**后累加即可
    
        import requests
        from bs4 import BeautifulSoup
        url='http://glidedsky.com/level/web/crawler-basic-1'
        headers={ 
            'Host':'glidedsky.com',
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Language':'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
            'Accept-Encoding':'gzip, deflate',
            'Referer':'http://glidedsky.com/level/crawler-basic-1',
            'Connection':'keep-alive',
            'Cookie':'XSRF-TOKEN=eyJpdiI6ImNrWHNhYk5aQzNrdXBiQ0pSY2dNakE9PSIsInZhbHVlIjoiZHRzeGkzZ1wvaUpoOXM2VVVralpzbFB4bXh6OVdIWnNad2txXC9jWFB0ZFA3dDlUY2Vvcis2RnJ3WlFqRUhqOXI3IiwibWFjIjoiNDI1NWYxZGVhMjUxNTZmOTNmZDQ1MmQ4YzYzMDIwYTllNTc5YzI2ZmIyMTMzN2IyZGRlMmRmMGIwYTVkZmU3ZiJ9; glidedsky_session=eyJpdiI6InNJZ1J4WGplOUNXbFlqU29ndzc4WFE9PSIsInZhbHVlIjoiSVdhY3ZQOCt1Z0psVWNDR0dLSzAyaVg3NHdBUndWZTJmUExmekpTOEUrNDhqXC9GVWFvVVhCSEwzR1REbm54WWsiLCJtYWMiOiJhZTQ2OGE1OTVmYTlkODViZGI0NDEyNDNiZmJmOWI3YzE3Y2I3Y2Q5OGY5M2ZlMjY5Y2U4NDU3MWIzODliOGMxIn0%3D; footprints=eyJpdiI6ImpuR2xMRVRualwvS0lhS2NtTTFLOWJBPT0iLCJ2YWx1ZSI6IlwvUEYrWWJnRWRCQ09mS2ZXY2EwZ0FodFFXdUNsTkVSZHRmUWxoZ0xIQXBLSHQwMGpsenZISFVXRkxnandydkZLIiwibWFjIjoiZWRhMzU2NTU0ZmRlNzMxM2ZhOWVmN2VlMzBhODNiNGM2MmMzMWMxYzIyZGFhMjQzMzcyODVmZjFmYTdiZDc4ZCJ9; Hm_lvt_020fbaad6104bcddd1db12d6b78812f6=1573543327; Hm_lpvt_020fbaad6104bcddd1db12d6b78812f6=1573548587; _ga=GA1.2.1195179388.1573543328; _gid=GA1.2.165778833.1573543328',
            'Upgrade-Insecure-Requests':'1',
            'Pragma':'no-cache',
            'Cache-Control':'no-cache'
        }
        response=requests.get(url,headers=headers)
        #生成解析器对象
        soup=BeautifulSoup(response.text,'lxml')
        #根据标签的特征获取所有满足特征的标签
        divs=soup.find_all('div',attrs={ 'class':'col-md-1'})
        #换成列表推导式
        nums=[int(div.text.strip()) for div in divs]
        sum_num=sum(nums)
        print(sum_num)
 *  结果  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 10]
 *  提交成功  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 11]

# GitHub源码：[https://github.com/coderyyn/spider\_example][https_github.com_coderyyn_spider_example] #

## 相关文章 ##

*  [Windows10下安装Anaconda3(64位)详细过程][Windows10_Anaconda3_64]
 *  [Windows下PyCharm的安装与使用][Windows_PyCharm]

我的个人博客网站是：[www.coderyyn.cn][]  
上面会不定期分享有关**爬虫、算法、环境搭建**以及有趣的帖子  
欢迎大家一起交流学习

**转载请注明**

[http_glidedsky.com_login]: http://glidedsky.com/login
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70]: /images/20230808/285a7b47480c468f99156c7487e3811e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 1]: /images/20230808/dee23eb4463e49f48941121a74e7a52c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 2]: /images/20230808/146619fe403547f7961473fcfb8ff194.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 3]: /images/20230808/fba03895c32f482e9f4278fb58b5c567.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 4]: /images/20230808/d4d6c0b2690d4f52b9d69cf350734d93.png
[HTTP]: https://blog.csdn.net/ychgyyn/article/details/102801882
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 5]: /images/20230808/cca3d5b9f5fa4b4ea61f75a593427b0b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 6]: /images/20230808/9c1dfa9fcf334d7fb4a7224e9a4f95cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 7]: /images/20230808/98d75e7766cc4d0a9cc7ef926bb83a48.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 8]: /images/20230808/27b85762f1fa49e886e86273b80e3a7d.png
[2019111217160721.png]: /images/20230808/2ada2f32bc634c9d96e59691de99b27b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 9]: /images/20230808/da83de0712144c9280922ad68b083b28.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 10]: /images/20230808/ceef4726f01e4fc2947a34b7e173816c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ljaGd5eW4_size_16_color_FFFFFF_t_70 11]: /images/20230808/5b9daadac79142ce8ce2e71bb637565e.png
[https_github.com_coderyyn_spider_example]: https://github.com/coderyyn/spider_example
[Windows10_Anaconda3_64]: https://blog.csdn.net/ychgyyn/article/details/82119201
[Windows_PyCharm]: https://blog.csdn.net/ychgyyn/article/details/96600396
[www.coderyyn.cn]: http:www.coderyyn.cn