1.背景

爬虫模拟浏览器获取网页的数据。
网页有唯一的URL地址，页面信息由HTML来描述，并使用HTTP/HTTPS协议来传输HTML网页数据；
关于：HTML（HyperText Markup Language）超文本标记语言是一种用于创建网页的标准标记语言。

2.安装

pip install bs4

C:\Users\G7>python -V
Python 3.6.8
C:\Users\G7>pip install bs4
Collecting bs4
  Downloading bs4-0.0.1.tar.gz (1.1 kB)
Collecting beautifulsoup4
  Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
     |████████████████████████████████| 115 kB 285 kB/s
Collecting soupsieve>1.2; python_version >= "3.0"
  Downloading soupsieve-2.0.1-py3-none-any.whl (32 kB)
Building wheels for collected packages: bs4
  Building wheel for bs4 (setup.py) ... done
  Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1277 sha256=5fd8e30825a0170b1f16f78e84fba037685af463843685dae9c0593bc7a8c926
  Stored in directory: c:\users\g7\appdata\local\pip\cache\wheels\19\f5\6d\a97dd4f22376d4472d5f4c76c7646876052ff3166b3cf71050
Successfully built bs4
Installing collected packages: soupsieve, beautifulsoup4, bs4
Successfully installed beautifulsoup4-4.9.3 bs4-0.0.1 soupsieve-2.0.1

3.简单示例

import requests
import bs4
if __name__=="__main__":
    url="http://www.cwl.gov.cn/"
    #构造一个向服务器请求资源的url对象
    req=requests.get(url=url)
    #设置请求网页的编码格式
    req.encoding="utf-8"
    #获得网页的源代码
    html=req.text
    #html parser解析器
    #BeautifulSoup 是一个可以从HTML或XML文件中提取数据的Python库
    bf=bs4.BeautifulSoup(html, "lxml")
    #遍历文档进行搜索
    img_url=bf.find_all(class_="n_item")[0]
    print(img_url)
    #<div class="n_item"><a href="/zzjg/" target="_blank">组织机构</a></div>