Pyspider使用过程教程以及若干问题记录

谁借莪１个温暖的怀抱￠ 2022-05-16 00:20 330阅读 0赞

## 使用教程 ##

![这里写图片描述][70]

![这里写图片描述][70 1]

![这里写图片描述][70 2]

![这里写图片描述][70 3]

![这里写图片描述][70 4]

![这里写图片描述][70 5]

![这里写图片描述][70 6]

![这里写图片描述][70 7]

4, pyspider中的操作界面截图 Case2  
代码示例如下：

#!/usr/bin/env python
    # -*- encoding: utf-8 -*-
    # Created on 2018-08-13 13:22:16
    # Project: repospider
    
    from pyspider.libs.base_handler import *
    
    
    class Handler(BaseHandler):
        crawl_config = {
        }
    
        @every(minutes=24 * 60)
        def on_start(self):
            self.crawl('http://www.reeoo.com', callback=self.index_page)
    
        @config(age=10 * 24 * 60 * 60)
        def index_page(self, response):
            for each in response.doc('div[class="thumb"]').items():
    
                detail_url = each('a').attr('href')
                print(detail_url)
                self.crawl(detail_url, callback=self.detail_page)
    
        @config(priority=2)
        def detail_page(self, response):
            header = response.doc('body > article > section > header')
            title = header('h1').text()
    
            tags = []
            for each in header.items('a'):
                tags.append(each.text())
    
            content = response.doc('div[id="post_content"]')
            description = content('blockquote > p').text()
    
            website_url = content('a').attr.href
    
            image_url_list = []
            for each in content.items('img[data-src]'):
               image_url_list.append(each.attr('data-src'))
    
            return {
                "title": title,
                "tags": tags,
                "description": description,
                "image_url_list": image_url_list,
                "website_url": website_url
                }

运行的界面截图：  
![这里写图片描述][70 8]

![这里写图片描述][70 9]

![这里写图片描述][70 10]

## 问题记录 ##

1.  问题1  
    问题的错误信息：

Exception: HTTP 599: Unable to communicate securely with peer: requested domain name does not match the server's certificate.

解决的办法：将代码中的基于https开头的地址，切换为http即可。实际的url地址还是https。  
办法2： 设置crawl的参数 validate\_cert=False

1.  使用Pyspider过程中的nginx代理  
    将Nginx的端口直接映射到web服务的端口：  
    
    
    > vim /etc/nginx/nginx/conf

文件的内容修改如下：  
“\`  
server \{  
listen 8080;  
server\_name localhost;  
root /usr/share/nginx/html;

location  / {
             proxy_pass http://127.0.0.1:5000;
         }
    
        error_page 404 /404.html;
            location = /40x.html {
        }
    
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

“\` 将nginx代理的主机8080映射为内部的5000端口。

1.  pyspider中的操作界面截图 Case1  
    case1:

#!/usr/bin/env python
    # -*- encoding: utf-8 -*-
    # Created on 2018-08-13 14:05:23
    # Project: test
    
    from pyspider.libs.base_handler import *
    
    
    class Handler(BaseHandler):
        crawl_config = {
        }
    
        @every(minutes=24 * 60)
        def on_start(self):
            self.crawl('http://reeoo.com/', callback=self.index_page)
    
        @config(age=10 * 24 * 60 * 60)
        def index_page(self, response):
            for each in response.doc('a[href^="http"]').items():
                self.crawl(each.attr.href, callback=self.detail_page)
    
        @config(priority=2)
        def detail_page(self, response):
            return {
                "url": response.url,
                "title": response.doc('title').text(),
            }

## 参考资料 ##

1.  [https://blog.csdn.net/weixin\_37947156/article/details/76495144][https_blog.csdn.net_weixin_37947156_article_details_76495144]

[70]: /images/20220516/717e2498774340adb899f002d624512c.png
[70 1]: /images/20220516/08537ca6f8f342b89e038f3279f7234a.png
[70 2]: /images/20220516/79d9230a9ec5489b904fc118b35c4823.png
[70 3]: /images/20220516/24f1b4196650448988e071d6237be08d.png
[70 4]: /images/20220516/b77ce99fbefa497186aa2583865c8dfe.png
[70 5]: /images/20220516/302a848e9d2c47fab7a98d8035970895.png
[70 6]: /images/20220516/7a41081856014f36af05b1d8e72ff89d.png
[70 7]: /images/20220516/d5f046db7caa4d40b2c658b13b9e43f5.png
[70 8]: /images/20220516/ea0609240fb74efeaada96a02ff3395d.png
[70 9]: /images/20220516/189d9239e8e64ca8a43b569a46377471.png
[70 10]: /images/20220516/525bfeb2d3944f60b9ece803439e2cb3.png
[https_blog.csdn.net_weixin_37947156_article_details_76495144]: https://blog.csdn.net/weixin_37947156/article/details/76495144