爬虫笔记：BeautifulSoup详解

墨蓝 2023-01-11 09:22 142阅读 0赞

## BeautifulSoup ##

Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。  
Beautiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时，Beautiful Soup就不能自动识别编码方式了。然后，你仅仅需要说明一下原始编码方式就可以了。  
Beautiful Soup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度。

## 用法详解 ##

<table> 
 <thead> 
  <tr> 
   <th>解析器</th> 
   <th>使用方法</th> 
   <th>优势</th> 
   <th>劣势</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Python标准库</td> 
   <td>BeautifulSoup(markup, “html.parser”)</td> 
   <td>Python的内置标准库、执行速度适中 、文档容错能力强</td> 
   <td>Python 2.7.3 or 3.2.2)前的版本中文容错能力差</td> 
  </tr> 
  <tr> 
   <td>lxml HTML 解析器</td> 
   <td>BeautifulSoup(markup, “lxml”)</td> 
   <td>速度快、文档容错能力强</td> 
   <td>需要安装C语言库</td> 
  </tr> 
  <tr> 
   <td>lxml XML 解析器</td> 
   <td>BeautifulSoup(markup, “xml”)</td> 
   <td>速度快、唯一支持XML的解析器</td> 
   <td>需要安装C语言库</td> 
  </tr> 
  <tr> 
   <td>html5lib</td> 
   <td>BeautifulSoup(markup, “html5lib”)</td> 
   <td>最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档</td> 
   <td>速度慢、不依赖外部扩展</td> 
  </tr> 
 </tbody> 
</table>

## **基本使用** ##

html = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title" name="dromouse"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"></a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')#使用lxml解析库
    print(soup.prettify())#格式化代码，自动补全代码。前面给的html不完整，有的只有头标签
    print(soup.title.string)#打印内容

prettify美化。前面给的html不全，有的不完整，只有头标签，soup.prettify()补全代码。soup.title.string 打印内容  
![20210126195635507.png_pic_center][]

**标签选择器**  
选择元素

html = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title" name="dromouse"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"></a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')#选择解析器
    print(soup.title)#选择title标签。标签即<>
    print(type(soup.title))
    print(soup.head)#选择head标签
    print(soup.p)#选择p标签

代码里p标签比较特殊，有多个，结果只输出了第一个p标签。意味着这种选择方式只会返回一个（第一个）  
![20210126200341414.png_pic_center][]  
获取标签名称

结果把标签名字输出来啦  
![20210126200725912.png_pic_center][]

获取标签属性

html = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title" name="dromouse"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"></a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.p.attrs['name'])#获取p标签name属性值
    print(soup.p['name'])#获取p标签name属性值

获取p标签name属性值  
![20210126201239513.png_pic_center][]

获取标签里的内容

html = """ <html><head><title>The Dormouse's story</title></head> <body> <p clss="title" name="dromouse"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"></a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.p.string)#获取p标签内容，该方法只能获取第一个p标签内容

获取p标签内容，该方法只能获取第一个p标签内容  
![20210126201446598.png_pic_center][]

标签嵌套选择

html = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title" name="dromouse"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"></a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.head.title.string)#嵌套选择,用点号分割，层层嵌套

![20210126201635596.png_pic_center][]

子节点和子孙节点

html = """ <html> <head> <title>The Dormouse's story</title> </head> <body> <p class="story"> Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"> <span>Elsie</span> </a> <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a> and they lived at the bottom of a well. </p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.p.contents)#获取p标签的子节点及其内容

print(soup.p.contents)contents\#获取标签的子孙节点及其内容。结果是列表形式  
![20210126202142412.png_pic_center][]

print(soup.p.children) children获取标签的子节点。返回结果是一个迭代器  
![20210126202456809.png_pic_center][]

descendants\#返回子孙节点（不仅仅是子节点，返回结果为迭代器）  
![2021012620275279.png_pic_center][]

父节点和祖先节点

soup.a.parent 返回的是第一个a标签的父节点  
![20210127092444501.png_pic_center][]

soup.a.parents 返回的是父节点以及祖先节点  
![20210127092627684.png_pic_center][]

兄弟节点

html = """ <html> <head> <title>The Dormouse's story</title> </head> <body> <p class="story"> Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1"> <span>Elsie</span> </a> <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a> and they lived at the bottom of a well. </p> <p class="story">...</p> """
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(list(enumerate(soup.a.next_siblings)))#a标签的下一个兄弟节点
    print(list(enumerate(soup.a.previous_siblings)))#上一个兄弟

sibling 翻译兄弟姐妹  
![2021012709304495.png_pic_center][]

**标准选择器find\_all**  
前面的选择方式是根据标签名来选择，但现实中不适合，因为一个html文档有很多相同名的标签。这时需要其他选择签  
find\_all( name , attrs , recursive , text , \*\*kwargs )  
可根据标签名、属性、内容查找文档

根据标签名name选择

find\_all(‘ul’) 找出ul标签。找出的是所有。返回的是列表形式  
![20210127093958714.png_pic_center][]

标签名name嵌套选择

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    for ul in soup.find_all('ul'):#因为返回结果是列表，所以可以遍历
        print(ul.find_all('li'))

![20210127094237550.png_pic_center][]

利用attrs选择

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1" name="elements"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.find_all(attrs={ 'id': 'list-1'}))#查找id为list-1的标签
    print('----')
    print(soup.find_all(attrs={ 'name': 'elements'}))

attrs接收的是一个字典型参数  
![20210127094734820.png_pic_center][]

可以不用attrs，直接在后面添加属性值

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.find_all(id='list-1'))
    print(soup.find_all(class_='element'))#因为classs是一个关键词，不能直接输入，所以在后面加一个下划线

![20210127094846871.png_pic_center][]

根据文本内容text来选择

返回的是内容。  
![20210127095248831.png_pic_center][]

标签选择器find  
find( name , attrs , recursive , text , \*\*kwargs )  
find返回单个元素，find\_all返回所有元素  
用法和findall一样

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.find('ul'))
    print(type(soup.find('ul')))
    print(soup.find('page'))

![20210127095924482.png_pic_center][]

find 其他用法

> > > **find\_parents() find\_parent()**  
> > > find\_parents()返回所有祖先节点，find\_parent()返回直接父节点。  
> > > \#%% md  
> > > **find\_next\_siblings() find\_next\_sibling()**  
> > > find\_next\_siblings()返回后面所有兄弟节点，find\_next\_sibling()返回后面第一个兄弟节点。  
> > > **find\_previous\_siblings() find\_previous\_sibling()**  
> > > find\_previous\_siblings()返回前面所有兄弟节点，find\_previous\_sibling()返回前面第一个兄弟节点。  
> > > **find\_all\_next() find\_next()**  
> > > find\_all\_next()返回节点后所有符合条件的节点, find\_next()返回第一个符合条件的节点  
> > > **find\_all\_previous() 和 find\_previous()**  
> > > find\_all\_previous()返回节点后所有符合条件的节点, find\_previous()返回第一个符合条件的节点

**css选择器**  
通过select()直接传入CSS选择器即可完成选择  
如果是class前加点,如果是id ，前面加\#

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    print(soup.select('.panel .panel-heading'))#嵌套选择class=panel后的class=panel-heading
    print(soup.select('ul li'))#嵌套选择ul标签下面的li标签
    print(soup.select('#list-2 .element'))#嵌套选择id=list-2,class=element
    print(type(soup.select('ul')[0]))#根据标签名选择

![20210127100817400.png_pic_center][]

标签选择前面不用加任何东西  
![20210127100926642.png_pic_center][]

css选择器获取属性  
用\[\]获取属性

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    for ul in soup.select('ul'):
        print(ul['id'])#获取属性id
        print(ul.attrs['id'])#获取属性id

![20210127101243109.png_pic_center][]

css选择器获取文本内容  
get\_text()获取文本内容

html=''' <div class="panel"> <div class="panel-heading"> <h4>Hello</h4> </div> <div class="panel-body"> <ul class="list" id="list-1"> <li class="element">Foo</li> <li class="element">Bar</li> <li class="element">Jay</li> </ul> <ul class="list list-small" id="list-2"> <li class="element">Foo</li> <li class="element">Bar</li> </ul> </div> </div> '''
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'lxml')
    for li in soup.select('li'):
        print(li.get_text())#get_text()获取文本内容

![20210127101510664.png_pic_center][]

## 总结 ##

*  推荐使用lxml解析库，必要时使用html.parser
 *  标签选择筛选功能弱但是速度快
 *  建议使用find()、find\_all() 查询匹配单个结果或者多个结果
 *  如果对CSS选择器熟悉建议使用select()
 *  记住常用的获取属性和文本值的方法

![在这里插入图片描述][20210127101620865.jpg_pic_center]  
作者：电气-余登武

[20210126195635507.png_pic_center]: /images/20221119/cfcff1c7737443cda8494f0498a8397c.png
[20210126200341414.png_pic_center]: /images/20221119/1aa2611e5f5d40898566f7ccc6c378e5.png
[20210126200725912.png_pic_center]: /images/20221119/99a917d590c24b76a1ca132f7f2207ac.png
[20210126201239513.png_pic_center]: /images/20221119/c00122b86b9b47e09b9aaf9389def456.png
[20210126201446598.png_pic_center]: /images/20221119/83cc1c656b99454a8547f821df3f9dd9.png
[20210126201635596.png_pic_center]: /images/20221119/97a7868dd68146f19840983d53880c0e.png
[20210126202142412.png_pic_center]: /images/20221119/d80c239c554943ce8df55915f192d62a.png
[20210126202456809.png_pic_center]: /images/20221119/67ac3a31c01145fe82cf747821d65ddb.png
[2021012620275279.png_pic_center]: /images/20221119/6473ed5a25024a57b66aaf7369bb3181.png
[20210127092444501.png_pic_center]: /images/20221119/f492be0d0573464880f9767bd33256c6.png
[20210127092627684.png_pic_center]: /images/20221119/4613eedaafd4460da912f551c75bcc2c.png
[2021012709304495.png_pic_center]: /images/20221119/2307c7719b1b4247a43a2a0762928b35.png
[20210127093958714.png_pic_center]: /images/20221119/d191c46b3c274875a17e0dc3cf6c809a.png
[20210127094237550.png_pic_center]: /images/20221119/600bed33e133401cb190da536943e9c7.png
[20210127094734820.png_pic_center]: https://img-blog.csdnimg.cn/20210127094734820.png#pic_center
[20210127094846871.png_pic_center]: https://img-blog.csdnimg.cn/20210127094846871.png#pic_center
[20210127095248831.png_pic_center]: https://img-blog.csdnimg.cn/20210127095248831.png#pic_center
[20210127095924482.png_pic_center]: https://img-blog.csdnimg.cn/20210127095924482.png#pic_center
[20210127100817400.png_pic_center]: https://img-blog.csdnimg.cn/20210127100817400.png#pic_center
[20210127100926642.png_pic_center]: https://img-blog.csdnimg.cn/20210127100926642.png#pic_center
[20210127101243109.png_pic_center]: https://img-blog.csdnimg.cn/20210127101243109.png#pic_center
[20210127101510664.png_pic_center]: https://img-blog.csdnimg.cn/20210127101510664.png#pic_center
[20210127101620865.jpg_pic_center]: https://img-blog.csdnimg.cn/20210127101620865.jpg#pic_center