A 2. 笔记 - scrapy创建项目与爬虫-嵩天（7.11--7.12）

悠悠 2022-05-20 04:23 133阅读 0赞

# 一、Scrapy爬虫的使用 #

## 1、流程 ##

1.  建立新工程：  
    a. cd 至目标地址  
    b. 输入命令`scrapy startproject python123demo`
2.  新建爬虫：  
    a. cd 至 python123demo 文件中  
    b. 输入命令`scrapy genspider demo python123.io` 创建名为demo的蜘蛛，域名为python123.io
3.  修改spider  
    a. 打开刚刚创建的py文件  
    b. `allowed_domains = ['python123.io']`可以删掉  
    c. 修改`start_urls = ['http://python123.io/']`中的url  
    d. 编辑parse(self, response)类，处理、存储
4.  运行爬虫  
    a. 输入`scrapy crawl demo`

注 yield：用生成器节省空间

## 2、request类 ##

![这里写图片描述][70]

![这里写图片描述][70 1]

## 3、Response类 ##

![这里写图片描述][70 2]  
![Response类型][Response]

## 4、CSS Selector 的基本使用 ##

## ![70 3][] ##

# 二、scrapy抓取股票 #

--------------------

## 1、功能设计与数据确定 ##

![这里写图片描述][70 4]

--------------------

![这里写图片描述][70 5]

--------------------

## 2、实证 ##

![这里写图片描述][70 6]

--------------------

![这里写图片描述][70 7]  
1. href 是Hypertext Reference的缩写。意思是指定超链接目标的URL。是css代码的一种。 \[etʃref\] 把h和ref拆开发音  
2. 处理了所有的股票信息，并把信息以字典的方式封装成Item给到了后面的pipelines

下面是stocks.py文件源代码：

# -*- coding: utf-8 -*-
    import scrapy
    import re
    
    class StocksSpider(scrapy.Spider):
        name = "stocks"
        start_urls = ['https://quote.eastmoney.com/stocklist.html']
    
        def parse(self, response):
            for href in response.css('a::attr(href)').extract():
                try:
                    stock = re.findall(r"[s][hz]\d{6}", href)[0]
                    url = 'https://gupiao.baidu.com/stock/' + stock + '.html'
                    yield scrapy.Request(url, callback=self.parse_stock)
                except:
                    continue
    
        def parse_stock(self, response):
            infoDict = {}
            stockInfo = response.css('.stock-bets')
            name = stockInfo.css('.bets-name').extract()[0]
            keyList = stockInfo.css('dt').extract()
            valueList = stockInfo.css('dd').extract()
            for i in range(len(keyList)):
                key = re.findall(r'>.*</dt>', keyList[i])[0][1:-5]
                try:
                    val = re.findall(r'\d+\.?.*</dd>', valueList[i])[0][0:-5]
                except:
                    val = '--'
                infoDict[key]=val
    
            infoDict.update(
                {
       '股票名称': re.findall('\s.*\(',name)[0].split()[0] + \
                 re.findall('\>.*\<', name)[0][1:-1]})
            yield infoDict

--------------------

![这里写图片描述][70 8]

1.  其中的每一个类都是对一个Item处理的过程

下面是pipelines.py文件源代码：

# -*- coding: utf-8 -*-
    
    # Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
    
    
    class BaidustocksPipeline(object):
        def process_item(self, item, spider):
            return item
    
    class BaidustocksInfoPipeline(object):
        def open_spider(self, spider):
            self.f = open('BaiduStockInfo.txt', 'w')
    
        def close_spider(self, spider):
            self.f.close()
    
        def process_item(self, item, spider):
            try:
                line = str(dict(item)) + '\n'
                self.f.write(line)
            except:
                pass
            return item

下面是settings.py文件中被修改的区域：

# Configure item pipelines
    # See https://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
    ITEM_PIPELINES = {
        'BaiduStocks.pipelines.BaidustocksInfoPipeline': 300,
    }

--------------------

[70]: /images/20220520/fb2bc9b10af041d8ac57c7887cca6175.png
[70 1]: /images/20220520/22cb19f3769c49aea0e4f30f673a84fa.png
[70 2]: /images/20220520/25a415d2d3f94341ae37ceab9afb93ff.png
[Response]: /images/20220520/8a5ab5b86a014e999e0d04d961163248.png
[70 3]: /images/20220520/5a5109b3bf6643a88750642824e6be32.png
[70 4]: /images/20220520/fe76a9f0884543588e95d8a0193267a5.png
[70 5]: /images/20220520/8cd0a96e09604b21b3e993482c0236a7.png
[70 6]: /images/20220520/8fb1385431bd4eef92acad5f60cfaa57.png
[70 7]: /images/20220520/8b87f253c28f42e9a3a19fc6bb04e4f8.png
[70 8]: /images/20220520/1ffd5009d0824f808137b064f9c9cf94.png