【Python爬虫】信息组织与提取方法

我会带着你远行 2023-02-24 08:45 66阅读 0赞

![format_png][]

![format_png 1][]

![format_png 2][]

![format_png 3][]

![format_png 4][]

![format_png 5][]

![format_png 6][]

![format_png 7][]![format_png 8][]

![format_png 9][]

![format_png 10][]

![format_png 11][]

![format_png 12][]

### 三种信息标记形式的比较 ###

![format_png 13][]

代码：

<person>
        <firstname>Tian</firstname>
        <lastname>Song</lastname>
        <address>
            <streetAddr>中关村南大街5号</streetAddr>
            <city>北京市</city>
            <zipcode>100081</zipcode>
        </address>
        <prof>Computer System</prof><prof>Security</prof>
    </person>

![format_png 14][]

代码：

{
        "firstName":"Tian",
        "lastName":"Song",
        "address":{
                    "streetAddr":"中关村南大街5号",
                    "city":"北京市",
                    "zipcode":"100081"
                   },
        "prof":["Computer System","Security"]
    }

![format_png 15][]

代码：

firstName: Tian
    lastName: Song
    address: 
        streetAddr: 中关村南大街5号
        city: 北京市
        zipcode: 100081
    prof:
    -Computer System
    -Security

![format_png 16][]

![format_png 17][]

### 信息提取的一般方法 ###

![format_png 18][]

![format_png 19][]

![format_png 20][]

![format_png 21][]

实践：

>>> import requests
    >>> r=requests.get("https://www.baidu.com/")
    >>> r.encoding='UTF-8'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> for link in soup.find_all('a'):
        print(link.get('href'))
    
        
    http://news.baidu.com
    https://www.hao123.com
    http://map.baidu.com
    http://v.baidu.com
    http://tieba.baidu.com
    http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1
    //www.baidu.com/more/
    http://home.baidu.com
    http://ir.baidu.com
    http://www.baidu.com/duty/
    http://jianyi.baidu.com/
    >>>

### 基于bs4库的HTML内容查找方法 ###

![format_png 22][]

练习：

>>> soup.find_all('a')
    [<a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a>, <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a>, <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a>, <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a>, <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a>, <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a>, <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a>, <a href="http://home.baidu.com">关于百度</a>, <a href="http://ir.baidu.com">About Baidu</a>, <a href="http://www.baidu.com/duty/">使用百度前必读</a>, <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a>]
    >>> soup.find_all(['a','b'])
    [<a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a>, <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a>, <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a>, <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a>, <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a>, <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a>, <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a>, <a href="http://home.baidu.com">关于百度</a>, <a href="http://ir.baidu.com">About Baidu</a>, <a href="http://www.baidu.com/duty/">使用百度前必读</a>, <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a>]
    >>> for tag in soup.find_all(True):
        print(tag.name)
    
        
    html
    head
    meta
    meta
    meta
    link
    title
    body
    div
    div
    div
    div
    div
    div
    img
    form
    input
    input
    input
    input
    input
    input
    span
    input
    span
    input
    div
    a
    a
    a
    a
    a
    noscript
    a
    script
    a
    div
    div
    p
    a
    a
    p
    a
    a
    img
    >>> import re
    >>> for tag in soup.find_all(re.compile('b')):
        print(tag.name)
    
        
    body
    >>>

![format_png 23][]

练习：

>>> soup.find_all('a','nav')
    []
    >>> soup.find_all('a','com')
    []
    >>> r.text
    '<!DOCTYPE html>\r\n<html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">登录</a>\');\r\n                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a>  <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号  <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n'
    >>> soup.find_all(id='wrapper')
    [<div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> <form action="//www.baidu.com/s" class="fm" id="form" name="f"> <input name="bdorz_come" type="hidden" value="1"/> <input name="ie" type="hidden" value="utf-8"/> <input name="f" type="hidden" value="8"/> <input name="rsv_bp" type="hidden" value="1"/> <input name="rsv_idx" type="hidden" value="1"/> <input name="tn" type="hidden" value="baidu"/><span class="bg s_ipt_wr"><input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/></span><span class="bg s_btn_wr"><input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/></span> </form> </div> </div> <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div> </div> </div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a href="http://home.baidu.com">关于百度</a> <a href="http://ir.baidu.com">About Baidu</a> </p> <p id="cp">©2017 Baidu <a href="http://www.baidu.com/duty/">使用百度前必读</a>  <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a> 京ICP证030173号  <img src="//www.baidu.com/img/gs.gif"/> </p> </div> </div> </div>]
    >>> soup.find_all('link')
    [<link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>]
    >>> import re
    >>> soup.find_all(id=re.compile('link'))
    []
    >>>

![format_png 24][]

练习：

![format_png 25][]

练习：

>>> soup
    <!DOCTYPE html>
    
    <html> <head><meta content="text/html;charset=utf-8" http-equiv="content-type"/><meta content="IE=Edge" http-equiv="X-UA-Compatible"/><meta content="always" name="referrer"/><link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/><title>百度一下，你就知道</title></head> <body link="#0000cc"> <div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> <form action="//www.baidu.com/s" class="fm" id="form" name="f"> <input name="bdorz_come" type="hidden" value="1"/> <input name="ie" type="hidden" value="utf-8"/> <input name="f" type="hidden" value="8"/> <input name="rsv_bp" type="hidden" value="1"/> <input name="rsv_idx" type="hidden" value="1"/> <input name="tn" type="hidden" value="baidu"/><span class="bg s_ipt_wr"><input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/></span><span class="bg s_btn_wr"><input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/></span> </form> </div> </div> <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div> </div> </div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a href="http://home.baidu.com">关于百度</a> <a href="http://ir.baidu.com">About Baidu</a> </p> <p id="cp">©2017 Baidu <a href="http://www.baidu.com/duty/">使用百度前必读</a>  <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a> 京ICP证030173号  <img src="//www.baidu.com/img/gs.gif"/> </p> </div> </div> </div> </body> </html>
    
    >>> soup.find_all(string="百度一下")
    []
    >>> import re
    >>> soup.find_all(string=re.compile("百度一下"))
    ['百度一下，你就知道']
    >>>

![format_png 26][]

![format_png 27][]

[format_png]: /images/20230209/0a22ca14312f4500822ca4f13523253e.png
[format_png 1]: /images/20230209/61fe04ef2a5046ee8a580f6be0f1db37.png
[format_png 2]: /images/20230209/500f8de7ed5c4c90ad86b6ac6ce9d7dc.png
[format_png 3]: /images/20230209/7b482c8250df4e96a8f52e76fee5902a.png
[format_png 4]: /images/20230209/35c62a4ab29745b088f55e5d43489306.png
[format_png 5]: /images/20230209/f1d826a9f0f248a79040d494a9d53283.png
[format_png 6]: /images/20230209/1c5f03da795a4af18de52f38194d5148.png
[format_png 7]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwODEzNTcxMjU3My0xNDk4Mzk5Nzg2LnBuZw?x-oss-process=image/format,png
[format_png 8]: /images/20230209/fe82049b07c243adb9bd779d096b4d33.png
[format_png 9]: /images/20230209/6185dade72b8469c810e72b9bf35b8c4.png
[format_png 10]: /images/20230209/86a9df6548384c62950d06a9f1a0e409.png
[format_png 11]: /images/20230209/e49f0f6b801d40fcbfa1e1513a53b4f2.png
[format_png 12]: /images/20230209/c51de858bdf9405eb89d8cbc0be7d768.png
[format_png 13]: /images/20230209/e5e90044fd62433e84bfc4509fce138b.png
[format_png 14]: /images/20230209/7cf4772b102f42dd9324ec4af9cf16ad.png
[format_png 15]: /images/20230209/9012c6a59ffe4d16b8e44d69281ec6e4.png
[format_png 16]: /images/20230209/619e0fc7d27949e895b59c9e255df581.png
[format_png 17]: /images/20230209/b24e82fa536a4f879a5a6c255a66ebc5.png
[format_png 18]: /images/20230209/f4d3b6d605d947e185f4822c604b8686.png
[format_png 19]: /images/20230209/c806b970947e48489b03a10b46ae7f01.png
[format_png 20]: /images/20230209/787fe8cb35604369ab1b57b29236d237.png
[format_png 21]: /images/20230209/e449c5b212874b45828336faea13f604.png
[format_png 22]: /images/20230209/711d8861bfe74c5798ec449c0d9107ed.png
[format_png 23]: /images/20230209/696cdc1589454b028ede2ebdfec74cce.png
[format_png 24]: /images/20230209/473304699da64537bbec0ed876a6f250.png
[format_png 25]: /images/20230209/b04fc1353e844e2186bda9fa964851fe.png
[format_png 26]: /images/20230209/6f07c11b98124401bdaceea9b248375f.png
[format_png 27]: /images/20230209/8a0d1cdec3e7495c9c40846f8044480f.png