运行scrapy 报错：CRITICAL: Unhandled error in Deferred [Errno 11] Resource temporarily unavailable

谁践踏了优雅 2023-06-15 07:21 20阅读 0赞

报错如下：

2019-11-21 03:56:07 [engine.py:256] INFO: Spider opened
    2019-11-21 03:56:07 [logstats.py:48] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    Unhandled error in Deferred:
    2019-11-21 03:56:09 [_legacy.py:154] CRITICAL: Unhandled error in Deferred:
    
    Traceback (most recent call last):
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/threading.py", line 916, in _bootstrap_inner
        self.run()
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/threading.py", line 864, in run
        self._target(*self._args, **self._kwargs)
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/_threads/_threadworker.py", line 46, in work
        task()
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/_threads/_team.py", line 190, in doWork
        task()
    --- <exception caught here> ---
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
        result = inContext.theWork()
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
        inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
        return func(*args,**kw)
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/site-packages/twisted/internet/_resolver.py", line 124, in get
        socketType)
      File "/home/ec2-user/anaconda3/envs/business_spider/lib/python3.6/socket.py", line 745, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    builtins.BlockingIOError: [Errno 11] Resource temporarily unavailable

原因：内置阻塞IO错误，资源暂时不可用，scrapy内部的原因，改一下启动的方式就好了：

之前的启动方式：

from twisted.internet import reactor, defer
    from scrapy.crawler import CrawlerRunner
    from scrapy.utils.log import configure_logging
    import time
    import logging
    from scrapy.utils.project import get_project_settings
    
    configure_logging()
    runner = CrawlerRunner(get_project_settings())
    
    
    @defer.inlineCallbacks
    def crawl():
        while True:
            logging.info("new cycle starting")
            yield runner.crawl('chinatest')
            time.sleep(300)
        reactor.stop()
    crawl()
    reactor.run()

改成如下，重启一下程序就不报错了。

import os
    import time
    
    
    while True:
        os.popen('scrapy crawl FinancialProducts_ForeignCurrency_Zhongxin')
        time.sleep(300)