python requests获取网页内容(urllib.request和requests的使用和区别)

末蓝、 2022-09-06 00:15 154阅读 0赞

urllib.request和requests的使用和区别   
请求网页内容可用urllib.request和requests函数,下面分别描述两种函数使用方法

**urllib.request的使用 **

1.发起基础请求

import urllib.request
    req = urllib.request.Request(url)
    response = urllib.request.urlopen(req)
    print(response)

返回结果如下:

![20210819102809181.png][]

返回的是一个object对象,如果想去获取内容还需进一步解析编码:

import urllib.request
    req = urllib.request.Request(url)
    response = urllib.request.urlopen(req)
    content  = response.read().decode("utf-8")
    print(content)

返回结果如下:

![20210819102826677.png][] 这个时候就是网页实际html内容,但是我们在获取网页内容,例如写爬虫时往往需要带上header头信息.   
2.发起带header的请求:

url = "https://movie.douban.com"
    headers = {
        "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    }
    
    req = urllib.request.Request(url=url,headers=headers)
    response = urllib.request.urlopen(req)
    content  = response.read().decode("utf-8")
    print(content)

有时我们还需要更多的参数 3.发起带有更多参数的请求头(以供参考):

url = "https://movie.douban.com"
    headers = {
        "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    }
    data = bytes(urllib.parse.urlencode({"hello":"world"}),encoding="utf-8")
    req = urllib.request.Request(url=url,data=data,headers=headers,method="POST")
    response = urllib.request.urlopen(req)
    content  = response.read().decode("utf-8")
    print(content)

**requests的使用 **  
  
requests的使用看起来就比较简单方便点 1.发起请求:

import requests
    url = "https://movie.douban.com"
    headers = {
        "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    }
    
    req = requests.get(url=url, headers=headers)
    print(req)

返回结果:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI3Njc3NjE_size_16_color_FFFFFF_t_70][]

返回的是一个状态码,这个时候requests提供一些方法来获取结果 1.req.text 得到的是str数据类型

返回结果:    
![20210819103021227.png][]  
2.req.content 得到的是Bytes类型,需要进行解码。

import requests
    url = "https://movie.douban.com"
    headers = {
        "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    }
    
     
    
    req = requests.get(url=url, headers=headers)
    print(req.content)

返回结果:    
![20210819103039331.png][]  
3.req.json 得到的是json数据

import requests
    url = "https://movie.douban.com"
    headers = {
        "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    }
    
     
    
    req = requests.get(url=url, headers=headers)
    print(req.json)

返回结果: ![20210819103431131.png][]

[20210819102809181.png]: /images/20220829/1c9d0312b1334949aea8df988707e87b.png
[20210819102826677.png]: /images/20220829/16c0dd246f8c45d2a20e759b0f6e3689.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI3Njc3NjE_size_16_color_FFFFFF_t_70]: /images/20220829/3c6a34c144b44e65b12b610e10c02dad.png
[20210819103021227.png]: /images/20220829/0bf0a4db55be4956a855565459a262e0.png
[20210819103039331.png]: /images/20220829/971db041e62f42738f2ef45b7d9097e8.png
[20210819103431131.png]: /images/20220829/f24d2e271d9340549b2a0c59498a7014.png