python中文爬取网页_使用Python爬虫爬取简单网页（Python爬虫入门）-Go语言中文社区...

我会带着你远行 2022-10-27 13:50 278阅读 0赞

今天我们来看一看使用Python爬取一些简单的网页。

所用工具：IDLE (Python 3.6 64-bit)

一. 爬取京东商品页面

![8d0a127b918867b8ebde6bf6dbda9f24.png][]

我将要爬取的是这个东京商品页面信息，代码如下：

import requests

url = "https://item.jd.com/6957643.html"

try:

r = requests.get(url)

r.raise\_for\_status()

r.encoding = r.apparent\_encoding

print(r.text\[:1000\])

except:

print("爬取失败")

二. 爬取亚马逊商品页面

![288cbe93e2c2add125ca92bfc6ffdf6e.png][]

我接下来要爬取这个亚马逊商品的页面，代码如下：

import requests

url = "https://www.amazon.cn/gp/product/B00W2T39C8/ref=cn\_ags\_s9\_asin?pf\_rd\_p=33e63d50-addd-4d44-a917-c9479c457e1a&pf\_rd\_s=merchandised-search-3&pf\_rd\_t=101&pf\_rd\_i=1403206071&pf\_rd\_m=A1AJ19PSB66TGU&pf\_rd\_r=FQQGZ7T42BF03V117HRD&pf\_rd\_r=FQQGZ7T42BF03V117HRD&pf\_rd\_p=33e63d50-addd-4d44-a917-c9479c457e1a&ref=cn\_ags\_s9\_asin\_1403206071\_merchandised-search-3"

try:

kv = \{'user-agent':'Mozilla/5.0'\}

r = requests.get(url,headers = kv)

r.raise\_for\_status()

r.encoding = r.apparent\_encoding

print(r.text\[1000:2000\])

except:

print("爬取失败")

三.  输入关键字爬取百度或360所能搜索到多少数据。代码如下：

import requests

keyword = "Python"

try:

kv = \{'wd':keyword\}\#如果用360就将键值对wd改成q将baidu改成so

r = requests.get("http://www.baidu.com/s",params = kv)

print(r.request.url)

r.raise\_for\_status()

print(len(r.text))

except:

print("爬取失败")

四. 爬取图片并存入指定地点(E://hh名字abc.jpg)。代码如下：

import requests

import os

url = "https://timgsa.baidu.com/timg?image&quality=80&size=b9999\_10000&sec=1533128040259&di=601acd33bcb188bfeb41cb50bc51ed41&imgtype=0&src=http%3A%2F%2Fs1.sinaimg.cn%2Fmw690%2F006LDoUHzy7auXElZGE40%26690"

path = "E://hh/abc.jpg"

try:

r = requests.get(url)

with open(path,'wb') as f:

f.write(r.content)

f.close()

print("文件已保存")

except :

print("爬取失败")

五.批量爬取图片

import requests

from bs4 import BeautifulSoup

import urllib.request

x = 0

def GetImg():

response = requests.get('http://www.mzitu.com/zipai/comment-page-2')

re = response.text

\#创建对象，解析网页

soup = BeautifulSoup(re,'html.parser')

\#找到img标签

girl = soup.find\_all('img')

for i in girl:

global x

imgl=i.get('src')

urllib.request.urlretrieve(imgl,'E:/python/xiuxiu/%s.jpg'%x)

x+=1

print("正在下载第%x张图片"%x)

def getHtml(url):

headers=\{'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0'\}

page1=urllib.request.Request(url,headers=headers)

page=urllib.request.urlopen(page1)

html=page.read()

GetImg()

感觉怎么样？学会了吗？

[8d0a127b918867b8ebde6bf6dbda9f24.png]: /images/20221024/3a4bae70b3f749cfa9725ad361082a60.png
[288cbe93e2c2add125ca92bfc6ffdf6e.png]: /images/20221024/8fdc83ffeb2b447aa0ed8e141305bf02.png