scrapy框架中多个spider,tiems,pipelines的使用及运行方法

朴灿烈づ我的快乐病毒、 2022-11-21 01:29 262阅读 0赞

#  #

用scrapy只创建一个项目，创建多个spider，每个spider指定items,pipelines.启动爬虫时只写一个启动脚本就可以全部同时启动。

本文代码已上传至github,链接在文未。

一，创建多个spider的scrapy项目

scrapy startproject mymultispider
    cd mymultispider
    scrapy genspider myspd1 sina.com.cn
    scrapy genspider myspd2 sina.com.cn
    scrapy genspider myspd3 sina.com.cn

二，运行方法

1.为了方便观察，在spider中分别打印相关信息

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

import scrapy
    class Myspd1Spider(scrapy.Spider):
        name = 'myspd1'
        allowed_domains = ['sina.com.cn']
        start_urls = ['http://sina.com.cn/']
    
    
        def parse(self, response):
            print('myspd1')

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

其他如myspd2,myspd3分别打印相关内容。

2.多个spider运行方法有两种，第一种写法比较简单，在项目目录下创建crawl.py文件，内容如下

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

from scrapy.crawler import CrawlerProcess
    from scrapy.utils.project import get_project_settings
    
    process = CrawlerProcess(get_project_settings())
    
    # myspd1是爬虫名
    process.crawl('myspd1')
    process.crawl('myspd2')
    process.crawl('myspd3')
    
    process.start()

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

为了观察方便，可在settings.py文件中限定日志输出

LOG_LEVEL = 'ERROR'

右键运行此文件即可，输出如下

![cd8554bb1df8f2f600544208411a966c.png][]

3.第二种运行方法为修改crawl源码，可以从官方的github中找到：[https://github.com/scrapy/scrapy/blob/master/scrapy/commands/crawl.py][https_github.com_scrapy_scrapy_blob_master_scrapy_commands_crawl.py]

在spiders目录的同级目录下创建一个mycmd目录，并在该目录中创建一个mycrawl.py,将crawl源码复制进来，修改其中的run方法，改为如下内容

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

def run(self, args, opts):
        # 获取爬虫列表
        spd_loader_list = self.crawler_process.spider_loader.list()
        # 遍历各爬虫
        for spname in spd_loader_list or args:
            self.crawler_process.crawl(spname, **opts.spargs)
            print("此时启动的爬虫：" + spname)
        self.crawler_process.start()

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**在该文件的目录下创建初始化文件\_\_init\_\_.py**

**完成后机构目录如下**

**![3bf74d9c1e2892436e41ccd35e901348.png][]**

使用命令启动爬虫

scrapy mycrawl --nolog

输出如下：

![d9eab6e78a9eafecd2e90471812274d5.png][]

三，指定items

1,这个比较简单，在items.py文件内创建相应的类，在spider中引入即可

items.py

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

import scrapy
    
    
    class MymultispiderItem(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        pass
    
    class Myspd1spiderItem(scrapy.Item):
        name = scrapy.Field()
    
    class Myspd2spiderItem(scrapy.Item):
        name = scrapy.Field()
    
    class Myspd3spiderItem(scrapy.Item):
        name = scrapy.Field()

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

spider内，例myspd1

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# -*- coding: utf-8 -*-
    import scrapy
    from mymultispider.items import Myspd1spiderItem
    
    class Myspd1Spider(scrapy.Spider):
        name = 'myspd1'
        allowed_domains = ['sina.com.cn']
        start_urls = ['http://sina.com.cn/']
    
        def parse(self, response):
            print('myspd1')
            item = Myspd1spiderItem()
            item['name'] = 'myspd1的pipelines'
            yield item

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

四，指定pipelines

1,这个也有两种方法，方法一，定义多个pipeline类：

pipelines.py文件内：

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

class Myspd1spiderPipeline(object):
        def process_item(self,item,spider):
            print(item['name'])
            return item
    
    class Myspd2spiderPipeline(object):
        def process_item(self,item,spider):
            print(item['name'])
            return item
    
    class Myspd3spiderPipeline(object):
        def process_item(self,item,spider):
            print(item['name'])
            return item

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

1.1settings.py文件开启管道

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

ITEM_PIPELINES = {
       # 'mymultispider.pipelines.MymultispiderPipeline': 300,
       'mymultispider.pipelines.Myspd1spiderPipeline': 300,
       'mymultispider.pipelines.Myspd2spiderPipeline': 300,
       'mymultispider.pipelines.Myspd3spiderPipeline': 300,
    }

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

1.2spider中设置管道，例myspd1

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# -*- coding: utf-8 -*-
    import scrapy
    from mymultispider.items import Myspd1spiderItem
    
    class Myspd1Spider(scrapy.Spider):
        name = 'myspd1'
        allowed_domains = ['sina.com.cn']
        start_urls = ['http://sina.com.cn/']
        custom_settings = {
            'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},
        }
    
        def parse(self, response):
            print('myspd1')
            item = Myspd1spiderItem()
            item['name'] = 'myspd1的pipelines'
            yield item

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

指定管道的代码

custom_settings = {
            'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},
        }

1.3运行crawl文件，运行结果如下

![c52bb17b94d39aff26c6823b04ce0760.png][]

2，方法二，在pipelines.py文件内判断是哪个爬虫的结果

2.1 pipelines.py文件内

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

class MymultispiderPipeline(object):
        def process_item(self, item, spider):
            if spider.name == 'myspd1':
                print('myspd1的pipelines')
            elif spider.name == 'myspd2':
                print('myspd2的pipelines')
            elif spider.name == 'myspd3':
                print('myspd3的pipelines')
            return item

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

2.2 settings.py文件内只开启MymultispiderPipeline这个管道文件

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

ITEM_PIPELINES = {
       'mymultispider.pipelines.MymultispiderPipeline': 300,
       # 'mymultispider.pipelines.Myspd1spiderPipeline': 300,
       # 'mymultispider.pipelines.Myspd2spiderPipeline': 300,
       # 'mymultispider.pipelines.Myspd3spiderPipeline': 300,
    }

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

2.3spider中屏蔽掉指定pipelines的相关代码

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# -*- coding: utf-8 -*-
    import scrapy
    from mymultispider.items import Myspd1spiderItem
    
    class Myspd1Spider(scrapy.Spider):
        name = 'myspd1'
        allowed_domains = ['sina.com.cn']
        start_urls = ['http://sina.com.cn/']
        # custom_settings = {
        #     'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},
        # }
    
        def parse(self, response):
            print('myspd1')
            item = Myspd1spiderItem()
            item['name'] = 'myspd1的pipelines'
            yield item

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

2.4 运行crawl.py文件，结果如下

![72649aafc38d87450a71c8fdc8b5ff81.png][]

代码git地址：[https://github.com/terroristhouse/crawler][https_github.com_terroristhouse_crawler]

python系列教程：

链接：https://pan.baidu.com/s/10eUCb1tD9GPuua5h\_ERjHA

提取码：h0td

[48304ba5e6f9fe08f3fa1abda7d326ab.png]: /images/20221120/2a707ffa0b744697b8a8bc53718b938e.png
[cd8554bb1df8f2f600544208411a966c.png]: https://img-blog.csdnimg.cn/img_convert/cd8554bb1df8f2f600544208411a966c.png
[https_github.com_scrapy_scrapy_blob_master_scrapy_commands_crawl.py]: https://github.com/scrapy/scrapy/blob/master/scrapy/commands/crawl.py
[3bf74d9c1e2892436e41ccd35e901348.png]: https://img-blog.csdnimg.cn/img_convert/3bf74d9c1e2892436e41ccd35e901348.png
[d9eab6e78a9eafecd2e90471812274d5.png]: https://img-blog.csdnimg.cn/img_convert/d9eab6e78a9eafecd2e90471812274d5.png
[c52bb17b94d39aff26c6823b04ce0760.png]: https://img-blog.csdnimg.cn/img_convert/c52bb17b94d39aff26c6823b04ce0760.png
[72649aafc38d87450a71c8fdc8b5ff81.png]: https://img-blog.csdnimg.cn/img_convert/72649aafc38d87450a71c8fdc8b5ff81.png
[https_github.com_terroristhouse_crawler]: https://github.com/terroristhouse/crawler