Python爬虫：十种网络请求异常情况示例

原创叁歲伎倆 2025-01-23 20:36 4阅读 0赞

在编写Python爬虫时，可能会遇到各种网络请求异常。以下列举了十种常见的异常及示例：

1. **HTTP状态码错误**：
   - 404 Not Found: 请求的资源不存在。
   ```python
   response = requests.get('http://nonexistent.example.com/')
   if response.status_code == 404:
       print('Resource not found.')
   ```

2. **网络连接问题**：
   - `requests.exceptions.ConnectionError`: 网络连接失败。
   ```python
   try:
       response = requests.get('https://www.example.com/', timeout=5) # 设置超时时间
       if response.status_code == 200:
           print('Successful connection.')
       else:
           print(f'Response status: {response.status_code}.')
     except ConnectionError as ce:
       print(ce)
   ```

3. **HTTP请求头问题**：
   - `requests.exceptions.InvalidHeader`： 请求头含有无效的值。
   ```python
   headers = {'User-Agent': 'Invalid User Agent'}
   try:
       response = requests.get('https://www.example.com/', headers=headers) 
       if response.status_code == 200:
           print('Successful connection.')
       else:
           print(f'Response status: {response.status_code}.')
     except InvalidHeader as ih:
       print(ih)
   ```

4. **代理设置问题**：
   - `requests.exceptions.ProxyError`： 使用的代理服务器出现问题。
   ```python
   proxy = 'http://proxy.example.com:8080/'
   try:
       response = requests.get('https://www.example.com/', proxies={'http': proxy}}) 
       if response.status_code == 200:
           print('Successful connection.')
       else:
           print(f'Response status: {response.status_code}.')
     except ProxyError as pe:
       print(pe)
   ```

5. **认证设置问题**：
   - `requests.exceptions.AuthenticationError`： 通过认证的请求失败。
   ```python
   username = 'your_username'
   password = 'your_password'
   try:
       response = requests.get('https://www.example.com/authenticated/', auth=(username, password)))) 
       if response.status_code == 200:
           print('Successful connection with authentication.')
       else:
           print(f'Response status: {response.status_code}.')
     except AuthenticationError as ae:
       print(ae)
   ```

6. **请求体问题**：
   - `requests.exceptions.MissingSchema`： 请求的URL缺少协议（如http或https）。
   ```python
   try:
       response = requests.get('example.com') # 缺少协议
       if response.status_code == 200:
           print('Successful connection without protocol.')
       else:
           print(f'Response status: {response.status_code}.')
     except MissingSchema as ms:
       print(ms)
   ```

7. **编码问题**：
   - `requests.exceptions.ChardetError`： 使用的字符集识别库出现错误。
   ```python
   try:
       response = requests.get('https://www.example.com/', encoding='latin1')) # 设置不正确的字符集
       if response.status_code == 200:
           print('Successful connection with incorrect encoding.')
       else:
           print(f'Response status: {response.status_code}.')
     except ChardetError as ce:
       print(ce)
   ```

8. **重试机制**：
   - `requests.exceptions.TooManyRedirects`： 操作导致超多重定向。
   ```python
   try:
       response = requests.get('https://www.example.com/redirected', allow_redirects=False)) # 关闭重定向
       if response.status_code == 200:
           print('Successful connection without redirects.')
       else:
           print(f'Response status: {response.status_code}.')
     except TooManyRedirects as trr:
       print(trr)
   ```

9. **超时问题**：
   - `requests.exceptions.Timeout`： 网络请求超时。
   ```python
   try:
       response = requests.get('https://www.example.com/slow', timeout=5)) # 设置超时时间
       if response.status_code == 200:
           print('Successful connection with timeout.')
       else:
           print(f'Response status: {response.status_code}.')
     except Timeout as to:
       print(to)
   ```

10. **响应数据问题**：
    - `requests.exceptions.ContentDecodingError`： 在解码响应内容时发生错误。
    ```python
    try:
       response = requests.get('https://www.example.com/json', encoding='utf-8')) # 设置编码，确保数据无误
       if response.status_code == 200:
           data = json.loads(response.text) # 解析JSON数据
           print(f'Response data: {data}}')
       else:
           print(f'Response status: {response.status_code}.'))
     except ContentDecodingError as cde:
       print(cde)
   ```

以上示例展示了十种网络请求异常情况，并通过Python的requests库进行模拟。在实际编写爬虫时，需要根据目标网站的具体情况进行调整和处理这些异常。