简单的爬行--静态网页爬虫+下一篇实例

墨蓝 2022-07-28 00:20 153阅读 0赞

![Center][]

![Center 1][]

![Center 2][]

![Center 3][]

![Center 4][]

![Center 5][]

![Center 6][]

![Center 7][]

1、有些网页需要用户登录才能访问，需要添加cookie的处理

2、有些网页需要代理才能访问

3、有些网页是使用HTTPS加密访问的

4、有些网页的URL存在相互自动的跳转关系

![Center 8][]

![Center 9][]

#coding=utf-8
 import urllib2
 
 url='https://www.baidu.com/'
 print '第一种方法'
 response1=urllib2.urlopen(url)
 print response1.getcode()#打印code看是否读取成功
 print len(response1.read())#打印下载网页的长度
 
 print '第二种方法'
 request=urllib2.Request(url)
 request.add_header("user-agent","Mozilla/5.0")#把URL伪装成了一个浏览器
 response2=urllib2.urlopen(request)
 print response2.getcode()#打印code看是否读取成功
 print len(response2.read())#打印下载网页的长度
 
 print '第三种方法'
 cj=cookielib.CookieJar()#创建一个cookie的容器
 opener=urllib2.build_opener(urllib2.HTTPCookiePrecessor(cj))
 urllib2.install_opener(opener)#给urllib2增加opener功能
 response3=urllib2.urlopen(url)
 print response3.getcode()#打印code看是否读取成功
 print cj
 print response3.read()#打印下载网页

![Center 10][]

![Center 11][]

![Center 12][]

![Center 13][]![Center 14][]

![Center 15][]

![Center 16][]

![Center 17][]

#coding=utf-8
 
 html_doc = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 The Dormouse's story
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 
 ...
 """
 
 from BeautifulSoup import BeautifulSoup
 soup=BeautifulSoup(html_doc,'html.parser',from_encoding='utf-8')
 
 
 print '获取所有的链接'
 links =soup.find_all('a')
 for link in links:
 print link.name,link['href'],link.get_text()
 
 
 print '获取lacie的链接'
 link_node=soup.find('a',href='http://example.com/lacie')
 print link_node.name,link_node['href'],link_node.get_text()
 
 print '正则匹配'
 link_node=soup.find('a',href=re.compile (r'ill'))
 print link_node.name,link_node['href'],link_node.get_text()
 
 print '获取P段落文字'
 p_node=soup.find('p',class_="title")
 print p_node.name,link_node.get_text()

【没运行出来，不知道问题在哪】

[Center]: /images/20220727/2d028d99080b43f99ff98fe5f956d4e5.png
[Center 1]: /images/20220727/0a840e74995a41b5a6326fa96f09babe.png
[Center 2]: /images/20220727/34ee73404ba244cab345cecbb8d9a916.png
[Center 3]: /images/20220727/b9895733ea1044b6b364d0d73ddc1531.png
[Center 4]: /images/20220727/834eadb2a2734ff9a2589d8750b881fe.png
[Center 5]: /images/20220727/b66f771346a24ad1ad994c0dd0ea27bf.png
[Center 6]: /images/20220727/691d7156abfa41fa80e24c01559ada10.png
[Center 7]: /images/20220727/0a5d036cedb34d84b73b7cd65df1d426.png
[Center 8]: /images/20220727/75679521fa6545d4aea8cf7da7b1d59a.png
[Center 9]: /images/20220727/1ec778b7d3634ba099b0aa8a51610613.png
[Center 10]: /images/20220727/6e03c50944ff482684556b32719a09bb.png
[Center 11]: /images/20220727/e0f2e620dc14477da842eba813adf370.png
[Center 12]: /images/20220727/7830ce974e8b4c02985099cb64334305.png
[Center 13]: https://img-blog.csdn.net/20160326161606084?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center
[Center 14]: /images/20220727/c07cf4ff6b2e42398a8b6b030d1f8e5c.png
[Center 15]: /images/20220727/6efe5d372e5844be969c1b81a1f01451.png
[Center 16]: /images/20220727/6fac1b8bc6584ab38d848c82a695c3ca.png
[Center 17]: /images/20220727/99d6239639394289bee7c2a45f286958.png