windows简易安装scrapy

超、凢脫俗 2022-05-19 12:58 220阅读 0赞

> Scrapy，Python开发的一个快速、高层次的屏幕抓取和web抓取框架，用于抓取web站点并从页面中提取结构化的数据。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。  
> Scrapy吸引人的地方在于它是一个框架，任何人都可以根据需求方便的修改。它也提供了多种类型爬虫的基类，如BaseSpider、sitemap爬虫等，最新版本又提供了web2.0爬虫的支持

写在前面：在本文中将使用的python版本为**3.7**，读者可自行选择版本。系统为**windows 64**位。

## 第一步 确保环境要求 ##

读者请自行安装python，并将python目录，以及python目录下Scripts加入系统环境变量中。如下图所示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JlbG9uZ2h1YW5nMTU3NDA1_size_16_color_FFFFFF_t_70]  
*注：在安装过程中如果读者勾选了将python加入环境变量，即跳过此步骤。*  
准备两个所需文件：**Twisted-18.7.0-cp37-cp37m-win\_amd64.whl  
lxml-4.2.3-cp37-cp37m-win\_amd64.whl**  
文件下载地址–>[https://pan.baidu.com/s/1TC2q\_oC5h6Z4ymRpmpSxsA][https_pan.baidu.com_s_1TC2q_oC5h6Z4ymRpmpSxsA] （包含了3.5 以及3.7版本）  
读者也可以自行下载–>非官方windows-python扩展包地址：pythonhttps://www.lfd.uci.edu/~gohlke/pythonlibs/  
*注：由于scrapy使用Twisted为框架，以及使用lxml解析html，在正常的安装过程中无法正确的安装这两个组件，故进行单独安装。*

## 第二步 安装scrapy ##

进入Twisted-18.7.0-cp37-cp37m-win\_amd64.whl 、lxml-4.2.3-cp37-cp37m-win\_amd64.whl文件存放目录，使用pip命令进行安装：

C:\Users\WU\Downloads\scrapyFile>pip install lxml-4.2.3-cp37-cp37m-win_amd64.whl
    
    C:\Users\WU\Downloads\scrapyFile>pip install Twisted-18.7.0-cp37-cp37m-win_amd64.whl

*注：本人将这两个文件存放C:\\Users\\WU\\Downloads\\scrapyFile文件中*  
在lxml Twisted安装成功后，执行如下命令，进行scrapy安装：

pip install pywin32
    pip install scrapy

*注：由于python后续还将访问windows系统的API库，故需安装pywin32*

## 第三步 验证scrapy是否安装成功 ##

在cmd中执行‘scrapy’，出现如下信息：

C:\Users\WU\Downloads\scrapyFile>scrapy
    Scrapy 1.5.1 - no active project
    
    Usage:
      scrapy <command> [options] [args]
    
    Available commands:
      bench         Run quick benchmark test
      fetch         Fetch a URL using the Scrapy downloader
      genspider     Generate new spider using pre-defined templates
      runspider     Run a self-contained spider (without creating a project)
      settings      Get settings values
      shell         Interactive scraping console
      startproject  Create new project
      version       Print Scrapy version
      view          Open URL in browser, as seen by Scrapy
    
      [ more ]      More commands available when run from project directory
    
    Use "scrapy <command> -h" to see more info about a command

## 第四步 创建scrapy项目 ##

进入工作目录，执行如下命令，即可查看对应项目的生成:

D:\pythonplace\scrapy>scrapy startproject helloworld
    New Scrapy project 'helloworld', using template directory 'd:\\software\\python3.7\\lib\\site-packages\\scrapy\\templates\\project', created in:
        D:\pythonplace\scrapy\helloworld
    
    You can start your first spider with:
        cd helloworld
        scrapy genspider example example.com

附录：  
1.当读者用运行scrapy crawl xxx命令启动爬虫时，出现如下错误：

Traceback (most recent call last):
      File "/usr/local/bin/scrapy", line 11, in <module>
        sys.exit(execute())
      File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 150, in execute
        _run_print_help(parser, _run_command, cmd, args, opts)
      File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
        func(*a, **kw)
      File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
        cmd.run(args, opts)
      File "/usr/local/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 57, in run
        self.crawler_process.crawl(spname, **opts.spargs)
      File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 170, in crawl
        crawler = self.create_crawler(crawler_or_spidercls)
      File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 198, in create_crawler
        return self._create_crawler(crawler_or_spidercls)
      File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 203, in _create_crawler
        return Crawler(spidercls, self.settings)
      File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 55, in __init__
        self.extensions = ExtensionManager.from_crawler(self)
      File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
        return cls.from_settings(crawler.settings, crawler)
      File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 34, in from_settings
        mwcls = load_object(clspath)
      File "/usr/local/lib/python3.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
        mod = import_module(module)
      File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/usr/local/lib/python3.7/site-packages/scrapy/extensions/telnet.py", line 12, in <module>
        from twisted.conch import manhole, telnet
      File "/usr/local/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
        def write(self, data, async=False):
                                  ^
    SyntaxError: invalid syntax

请找到python目录下Lib/site-packages/twisted/conch/manhole.py文件的154、155、240、241、247行的async重命名  
如下：

154    def write(self, data, async1=False):
    155        self.handler.addOutput(data, async1)
        
           ........
    
    240    def addOutput(self, data, async1=False):
    241        if async1:
           ........
    
    247        if async1:

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JlbG9uZ2h1YW5nMTU3NDA1_size_16_color_FFFFFF_t_70]: /images/20220519/5d726051c5804e2993d941827d65cc78.png
[https_pan.baidu.com_s_1TC2q_oC5h6Z4ymRpmpSxsA]: https://pan.baidu.com/s/1TC2q_oC5h6Z4ymRpmpSxsA