【Python爬虫】Beautiful Soup库入门

妖狐艹你老母 2023-02-23 09:30 40阅读 0赞

## BeautifulSoup库的安装 ##

![format_png][]

安装

pip install beautifulsoup4

![format_png 1][]

测试是否安装成功

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license()" for more information.
    >>> import requests
    >>> r=requests.get("https://www.baidu.com/")
    >>> r.text
    '<!DOCTYPE html>\r\n<html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>ç\x99¾åº¦ä¸\x80ä¸\x8bï¼\x8cä½\xa0å°±ç\x9f¥é\x81\x93</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=ç\x99¾åº¦ä¸\x80ä¸\x8b class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>æ\x96°é\x97»</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>å\x9c°å\x9b¾</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>è§\x86é¢\x91</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>è´´å\x90§</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>ç\x99»å½\x95</a> </noscript> <script>document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">ç\x99»å½\x95</a>\');\r\n                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">æ\x9b´å¤\x9aäº§å\x93\x81</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>å\x85³äº\x8eç\x99¾åº¦</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>ä½¿ç\x94¨ç\x99¾åº¦å\x89\x8då¿\x85è¯»</a>  <a href=http://jianyi.baidu.com/ class=cp-feedback>æ\x84\x8fè§\x81å\x8f\x8dé¦\x88</a> äº¬ICPè¯\x81030173å\x8f·  <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n'
    >>> r.encoding='UTF-8'
    >>> r.text
    '<!DOCTYPE html>\r\n<html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">登录</a>\');\r\n                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a>  <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号  <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> print(soup.prettify())
    
    >>>

![format_png 2][]

## Beautiful Soup库的基本元素 ##

![format_png 3][]

![format_png 4][]

![format_png 5][]

![format_png 6][]

![format_png 7][]

![format_png 8][]

练习记录

>>> import requests
    >>> r=requests.get("https://python123.io/ws/demo.html")
    >>> r.text
    '<html><head><title>This is a python demo page</title></head>\r\n<body>\r\n<p class="title"><b>The demo python introduces several python courses.</b></p>\r\n<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>\r\n</body></html>'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> print(soup.prettify())
    <html>
     <head>
      <title>
       This is a python demo page
      </title>
     </head>
     <body>
      <p class="title">
       <b>
        The demo python introduces several python courses.
       </b>
      </p>
      <p class="course">
       Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
       <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
        Basic Python
       </a>
       and
       <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
        Advanced Python
       </a>
       .
      </p>
     </body>
    </html>
    >>> from bs4 import BeautifulSoup
    >>> soup=Beautiful("<html>data</html>","html.parser")
    Traceback (most recent call last):
      File "<pyshell#8>", line 1, in <module>
        soup=Beautiful("<html>data</html>","html.parser")
    NameError: name 'Beautiful' is not defined
    >>> soup=BeautifulSoup("<html>data</html>","html.parser")
    >>> soup2=BeautifulSoup(open("D://MyProject//Python学习//爬虫学习//demo.html")
    4444
                
    SyntaxError: invalid syntax
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> soup.title
    <title>This is a python demo page</title>
    >>> tag=soup.a
    >>> tag
    <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> soup.a.name
    'a'
    >>> soup.a.paernt.name
    Traceback (most recent call last):
      File "<pyshell#23>", line 1, in <module>
        soup.a.paernt.name
    AttributeError: 'NoneType' object has no attribute 'name'
    >>> soup.a.parent.name
    'p'
    >>> soup.a.parent.parent.name
    'body'
    >>> tag=soup.a
    >>> tag.attrs
    {
        'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
    >>> tag.attrs['class']
    ['py1']
    >>> tag.attrs['href']
    'http://www.icourse163.org/course/BIT-268001'
    >>> type(tag.attrs)
    <class 'dict'>
    >>> type(tag)
    <class 'bs4.element.Tag'>
    >>> soup.a
    <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
    >>> soup.a.string
    'Basic Python'
    >>> soup.p
    <p class="title"><b>The demo python introduces several python courses.</b></p>
    >>> soup.p.string
    'The demo python introduces several python courses.'
    >>> type(soup.p.string)
    <class 'bs4.element.NavigableString'>
    >>> newsoup=BeautifulSoup("<b><!--This is a comment++></b><p>This is a comment</p>","html.parser")
    >>> newsoup.b.string
    '<!--This is a comment++>'
    >>> type(newsoup.b.string)
    <class 'bs4.element.NavigableString'>
    >>> newsoup.p.string
    'This is a comment'
    >>> type(newsoup.p.string)
    <class 'bs4.element.NavigableString'>
    >>>

![format_png 9][]

## 基于bs4库的HTML内容遍历方法 ##

![format_png 10][]

![format_png 11][]

![format_png 12][]

练习

>>> import requests
    >>> r=requests.get("https://www.baidu.com/")
    >>> r.text
    '<!DOCTYPE html>\r\n<html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>ç\x99¾åº¦ä¸\x80ä¸\x8bï¼\x8cä½\xa0å°±ç\x9f¥é\x81\x93</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=ç\x99¾åº¦ä¸\x80ä¸\x8b class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>æ\x96°é\x97»</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>å\x9c°å\x9b¾</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>è§\x86é¢\x91</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>è´´å\x90§</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>ç\x99»å½\x95</a> </noscript> <script>document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">ç\x99»å½\x95</a>\');\r\n                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">æ\x9b´å¤\x9aäº§å\x93\x81</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>å\x85³äº\x8eç\x99¾åº¦</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>ä½¿ç\x94¨ç\x99¾åº¦å\x89\x8då¿\x85è¯»</a>  <a href=http://jianyi.baidu.com/ class=cp-feedback>æ\x84\x8fè§\x81å\x8f\x8dé¦\x88</a> äº¬ICPè¯\x81030173å\x8f·  <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n'
    >>> r.encoding='UTF-8'
    >>> r.text
    '<!DOCTYPE html>\r\n<html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">登录</a>\');\r\n                </script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a>  <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号  <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\r\n'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> soup.head
    <head><meta content="text/html;charset=utf-8" http-equiv="content-type"/><meta content="IE=Edge" http-equiv="X-UA-Compatible"/><meta content="always" name="referrer"/><link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/><title>百度一下，你就知道</title></head>
    >>> soup.head.contents
    [<meta content="text/html;charset=utf-8" http-equiv="content-type"/>, <meta content="IE=Edge" http-equiv="X-UA-Compatible"/>, <meta content="always" name="referrer"/>, <link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>, <title>百度一下，你就知道</title>]
    >>> soup.body.contents
    [' ', <div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> <form action="//www.baidu.com/s" class="fm" id="form" name="f"> <input name="bdorz_come" type="hidden" value="1"/> <input name="ie" type="hidden" value="utf-8"/> <input name="f" type="hidden" value="8"/> <input name="rsv_bp" type="hidden" value="1"/> <input name="rsv_idx" type="hidden" value="1"/> <input name="tn" type="hidden" value="baidu"/><span class="bg s_ipt_wr"><input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/></span><span class="bg s_btn_wr"><input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/></span> </form> </div> </div> <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div> </div> </div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a href="http://home.baidu.com">关于百度</a> <a href="http://ir.baidu.com">About Baidu</a> </p> <p id="cp">©2017 Baidu <a href="http://www.baidu.com/duty/">使用百度前必读</a>  <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a> 京ICP证030173号  <img src="//www.baidu.com/img/gs.gif"/> </p> </div> </div> </div>, ' ']
    >>> len(soup.body.contents)
    3
    >>> soup.body.contents[1]
    <div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> <form action="//www.baidu.com/s" class="fm" id="form" name="f"> <input name="bdorz_come" type="hidden" value="1"/> <input name="ie" type="hidden" value="utf-8"/> <input name="f" type="hidden" value="8"/> <input name="rsv_bp" type="hidden" value="1"/> <input name="rsv_idx" type="hidden" value="1"/> <input name="tn" type="hidden" value="baidu"/><span class="bg s_ipt_wr"><input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/></span><span class="bg s_btn_wr"><input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/></span> </form> </div> </div> <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div> </div> </div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a href="http://home.baidu.com">关于百度</a> <a href="http://ir.baidu.com">About Baidu</a> </p> <p id="cp">©2017 Baidu <a href="http://www.baidu.com/duty/">使用百度前必读</a>  <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a> 京ICP证030173号  <img src="//www.baidu.com/img/gs.gif"/> </p> </div> </div> </div>
    >>>

![format_png 13][]

![format_png 14][]

练习

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license()" for more information.
    >>> import requests
    >>> r=requests.get("https://www.baidu.com/")
    >>> r.encoding='UTF-8'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> soup.title.parent
    <head><meta content="text/html;charset=utf-8" http-equiv="content-type"/><meta content="IE=Edge" http-equiv="X-UA-Compatible"/><meta content="always" name="referrer"/><link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/><title>百度一下，你就知道</title></head>
    >>> soup.html.parent
    <!DOCTYPE html>
    
    <html> <head><meta content="text/html;charset=utf-8" http-equiv="content-type"/><meta content="IE=Edge" http-equiv="X-UA-Compatible"/><meta content="always" name="referrer"/><link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/><title>百度一下，你就知道</title></head> <body link="#0000cc"> <div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> <form action="//www.baidu.com/s" class="fm" id="form" name="f"> <input name="bdorz_come" type="hidden" value="1"/> <input name="ie" type="hidden" value="utf-8"/> <input name="f" type="hidden" value="8"/> <input name="rsv_bp" type="hidden" value="1"/> <input name="rsv_idx" type="hidden" value="1"/> <input name="tn" type="hidden" value="baidu"/><span class="bg s_ipt_wr"><input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/></span><span class="bg s_btn_wr"><input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/></span> </form> </div> </div> <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div> </div> </div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a href="http://home.baidu.com">关于百度</a> <a href="http://ir.baidu.com">About Baidu</a> </p> <p id="cp">©2017 Baidu <a href="http://www.baidu.com/duty/">使用百度前必读</a>  <a class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈</a> 京ICP证030173号  <img src="//www.baidu.com/img/gs.gif"/> </p> </div> </div> </div> </body> </html>
    
    >>> soup.parent
    >>>

![format_png 15][]

实践

![format_png 16][]

![format_png 17][]

![format_png 18][]

练习：

>>> soup.a.next_sibling
    ' '
    >>> soup.a.next_sibling.next_sibling
    <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a>
    >>> soup.a.previous_sibling
    ' '
    >>> soup.a.previous_sibling.previous_sibling
    >>> soup.a.parent
    <div id="u1"> <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧</a> <noscript> <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
    
                    </script> <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品</a> </div>
    >>>

![format_png 19][]

## 基于bs4库的HTML格式化和编码 ##

![format_png 20][]

练习：

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license()" for more information.
    >>> import requests
    >>> r=requests.get("https://www.baidu.com/")
    >>> r.encoding='UTF-8'
    >>> demo=r.text
    >>> from bs4 import BeautifulSoup
    >>> soup=BeautifulSoup(demo,"html.parser")
    >>> soup.prettify()
    '<!DOCTYPE html>\n\n<html>\n <head>\n  <meta content="text/html;charset=utf-8" http-equiv="content-type"/>\n  <meta content="IE=Edge" http-equiv="X-UA-Compatible"/>\n  <meta content="always" name="referrer"/>\n  <link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>\n  <title>\n   百度一下，你就知道\n  </title>\n </head>\n <body link="#0000cc">\n  <div id="wrapper">\n   <div id="head">\n    <div class="head_wrapper">\n     <div class="s_form">\n      <div class="s_form_wrapper">\n       <div id="lg">\n        <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/>\n       </div>\n       <form action="//www.baidu.com/s" class="fm" id="form" name="f">\n        <input name="bdorz_come" type="hidden" value="1"/>\n        <input name="ie" type="hidden" value="utf-8"/>\n        <input name="f" type="hidden" value="8"/>\n        <input name="rsv_bp" type="hidden" value="1"/>\n        <input name="rsv_idx" type="hidden" value="1"/>\n        <input name="tn" type="hidden" value="baidu"/>\n        <span class="bg s_ipt_wr">\n         <input autocomplete="off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/>\n        </span>\n        <span class="bg s_btn_wr">\n         <input autofocus="" class="bg s_btn" id="su" type="submit" value="百度一下"/>\n        </span>\n       </form>\n      </div>\n     </div>\n     <div id="u1">\n      <a class="mnav" href="http://news.baidu.com" name="tj_trnews">\n       新闻\n      </a>\n      <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">\n       hao123\n      </a>\n      <a class="mnav" href="http://map.baidu.com" name="tj_trmap">\n       地图\n      </a>\n      <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">\n       视频\n      </a>\n      <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">\n       贴吧\n      </a>\n      <noscript>\n       <a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1" name="tj_login">\n        登录\n       </a>\n      </noscript>\n      <script>\n       document.write(\'<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\'+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ \'" name="tj_login" class="lb">登录</a>\');\n      </script>\n      <a class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">\n       更多产品\n      </a>\n     </div>\n    </div>\n   </div>\n   <div id="ftCon">\n    <div id="ftConw">\n     <p id="lh">\n      <a href="http://home.baidu.com">\n       关于百度\n      </a>\n      <a href="http://ir.baidu.com">\n       About Baidu\n      </a>\n     </p>\n     <p id="cp">\n      ©2017\xa0Baidu\n      <a href="http://www.baidu.com/duty/">\n       使用百度前必读\n      </a>\n      <a class="cp-feedback" href="http://jianyi.baidu.com/">\n       意见反馈\n      </a>\n      京ICP证030173号\n      <img src="//www.baidu.com/img/gs.gif"/>\n     </p>\n    </div>\n   </div>\n  </div>\n </body>\n</html>\n'
    >>> print(soup.prettify())

<!DOCTYPE html>  
  
<html>  
<head>  
<meta content="text/html;charset=utf-8" http-equiv="content-type"/>  
<meta content="IE=Edge" http-equiv="X-UA-Compatible"/>  
<meta content="always" name="referrer"/>  
<link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>  
<title>  
百度一下，你就知道  
</title>  
</head>  
<body link="\#0000cc">  
<div id="wrapper">  
<div id="head">  
<div class="head\_wrapper">  
<div class="s\_form">  
<div class="s\_form\_wrapper">  
<div id="lg">  
<img height="129" hidefocus="true" src="//www.baidu.com/img/bd\_logo1.png" width="270"/>  
</div>  
<form action="//www.baidu.com/s" class="fm" id="form" name="f">  
<input name="bdorz\_come" type="hidden" value="1"/>  
<input name="ie" type="hidden" value="utf-8"/>  
<input name="f" type="hidden" value="8"/>  
<input name="rsv\_bp" type="hidden" value="1"/>  
<input name="rsv\_idx" type="hidden" value="1"/>  
<input name="tn" type="hidden" value="baidu"/>  
<span class="bg s\_ipt\_wr">  
<input autocomplete="off" autofocus="autofocus" class="s\_ipt" id="kw" maxlength="255" name="wd" value=""/>  
</span>  
<span class="bg s\_btn\_wr">  
<input autofocus="" class="bg s\_btn" id="su" type="submit" value="百度一下"/>  
</span>  
</form>  
</div>  
</div>  
<div id="u1">  
<a class="mnav" href="http://news.baidu.com" name="tj\_trnews">  
新闻  
</a>  
<a class="mnav" href="https://www.hao123.com" name="tj\_trhao123">  
hao123  
</a>  
<a class="mnav" href="http://map.baidu.com" name="tj\_trmap">  
地图  
</a>  
<a class="mnav" href="http://v.baidu.com" name="tj\_trvideo">  
视频  
</a>  
<a class="mnav" href="http://tieba.baidu.com" name="tj\_trtieba">  
贴吧  
</a>  
<noscript>  
<a class="lb" href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz\_come%3d1" name="tj\_login">  
登录  
</a>  
</noscript>  
<script>  
document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz\_come=1")+ '" name="tj\_login" class="lb">登录</a>');  
</script>  
<a class="bri" href="//www.baidu.com/more/" name="tj\_briicon" style="display: block;">  
更多产品  
</a>  
</div>  
</div>  
</div>  
<div id="ftCon">  
<div id="ftConw">  
<p id="lh">  
<a href="http://home.baidu.com">  
关于百度  
</a>  
<a href="http://ir.baidu.com">  
About Baidu  
</a>  
</p>  
<p id="cp">  
©2017 Baidu  
<a href="http://www.baidu.com/duty/">  
使用百度前必读  
</a>  
<a class="cp-feedback" href="http://jianyi.baidu.com/">  
意见反馈  
</a>  
京ICP证030173号  
<img src="//www.baidu.com/img/gs.gif"/>  
</p>  
</div>  
</div>  
</div>  
</body>  
</html>

>>>

![format_png 21][]

编码要用“UTF-8”

![format_png 22][]

[format_png]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNTEyNDc0NjMwMS0yMDg4NTE0MTQ1LnBuZw?x-oss-process=image/format,png
[format_png 1]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNTEyNTMxMDcyMS02NzY4OTI5MTgucG5n?x-oss-process=image/format,png
[format_png 2]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNTEzMDUyOTE0Mi0yMzg4OTEzMDQucG5n?x-oss-process=image/format,png
[format_png 3]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDE1MzY4MS02NDk2Mjg0NDQucG5n?x-oss-process=image/format,png
[format_png 4]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDIyMjI4Ny03ODkzMzcwMDUucG5n?x-oss-process=image/format,png
[format_png 5]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDMyMzcwOC02ODg0NDA3NjMucG5n?x-oss-process=image/format,png
[format_png 6]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDUxMDgzNy0xNDA3NDE1MzM4LnBuZw?x-oss-process=image/format,png
[format_png 7]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDY0NjYzOS0xNjEzNzA2MjY1LnBuZw?x-oss-process=image/format,png
[format_png 8]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjA5NDcyNDI2NS0xMTUwOTM4NjA5LnBuZw?x-oss-process=image/format,png
[format_png 9]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjEwNDc0NTE0MS0yMDU2NjAyNjk4LnBuZw?x-oss-process=image/format,png
[format_png 10]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjEwNTIwMjMwNS0xNzQyMDUwODA0LnBuZw?x-oss-process=image/format,png
[format_png 11]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjEwNTQ0MTc1Ni0yMjA4NjM0NjAucG5n?x-oss-process=image/format,png
[format_png 12]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjEwNTQ1Nzc4OS02MzUxODEzMjYucG5n?x-oss-process=image/format,png
[format_png 13]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMTAyMDkyNC0xMTYzOTg2NTAzLnBuZw?x-oss-process=image/format,png
[format_png 14]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMTE0MjY3OS0xNzcxNTgzNDIucG5n?x-oss-process=image/format,png
[format_png 15]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMTYxMTAxMi0xMTM4NTczNTc1LnBuZw?x-oss-process=image/format,png
[format_png 16]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMTkwODgxNC05NDI3NzAyNzEucG5n?x-oss-process=image/format,png
[format_png 17]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMTk1NjQ2NS04NjY3Mzg5Ni5wbmc?x-oss-process=image/format,png
[format_png 18]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMjExMzA1Ni0xNTM0NzczMjMzLnBuZw?x-oss-process=image/format,png
[format_png 19]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMjYzODc3NC05MzAyMzk4MDcucG5n?x-oss-process=image/format,png
[format_png 20]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMjkyODI2Ni0yNzg1ODM5ODIucG5n?x-oss-process=image/format,png
[format_png 21]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMzMxOTg0OC0xNTIxMzE5MDU5LnBuZw?x-oss-process=image/format,png
[format_png 22]: http://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDIwLmNuYmxvZ3MuY29tL2Jsb2cvMTc4MzAzMC8yMDIwMDcvMTc4MzAzMC0yMDIwMDcwNjExMzUzODM3OS0xOTQyNTAxNjY0LnBuZw?x-oss-process=image/format,png