java——强大的jsoup

偏执的太偏执、 2022-06-15 00:14 167阅读 0赞

# java——强大的jsoup #

## 1、jsoup简介和功能 ##

jsoup 是一款 Java 的 HTML 解析器，可直接解析某个 URL 地址、HTML 文本内容。它提供了一套非常省力的 API，可通过 DOM，CSS 以及类似于 jQuery 的操作方法来取出和操作数据。

jsoup的几个强大的功能：

> 1. 从一个 URL，文件或字符串中解析 HTML；
> 
> 2. 使用 DOM 或 CSS 选择器来查找、取出数据；
> 
> 3. 可操作 HTML 元素、属性、文本；
> 
> jsoup下载：[jsoup.jar][]

## 2、加载HTML文档 ##

jsoup 可以从包括字符串、URL 地址以及本地文件来加载 HTML 文档，并生成 Document 对象实例。

1、从字符串中加载HTML文档

// 直接从字符串中输入 HTML 文档
    		String html = "<html><head><title>jsoup读入文档</title></head>" + "<body><p>jsoup使用字符串读入文档</p></body></html>";
    		Document doc = Jsoup.parse(html);
    		System.out.println(doc);
    		String title = doc.title();
    		System.out.println(title);

2、通过URL加载HTML文档

// 从 URL 直接加载 HTML 文档
    		try {
    			Document doc1 = Jsoup.connect("http://www.baidu.com").get();
    			title = doc1.title();
    			System.out.println(doc1);
    			System.out.println(title);
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}

try {
    			Document doc2 = Jsoup.connect("http://www.oschina.net/") 
    					  .data("query", "Java")   // 请求参数
    					  .userAgent("I ’ m jsoup") // 设置 User-Agent 
    					  .cookie("auth", "token") // 设置 cookie 
    					  .timeout(3000)           // 设置连接超时时间
    					  .post();                 // 使用 POST 方法访问 URL 
    			System.out.println(doc2);
    			title = doc2.title();
    			System.out.println(title);
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}                 // 使用 POST 方法访问 URL

3、通过本地文件加载HTML文件

// 从文件中加载 HTML 文档
    		 File input = new File("test.html"); 
    		 try {
    			Document doc3 = Jsoup.parse(input,"UTF-8","http://www.oschina.net/");
    			System.out.println(doc3);
    			title = doc3.title();
    			System.out.println(title);
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}

其中，parse方法中第三个参数的功能为：当 HTML 文档使用相对路径方式引用外部文件时，jsoup 会自动为这些 URL 加上一个前缀，也就是这个 baseURL。例如 <a href=/project> 开源软件 </a> 会被转换成 <a href=http://www.oschina.net/project> 开源软件 </a>。

## 3、初步解析HTML页面数据 ##

try {
    			Document doc = Jsoup.connect("http://www.baidu.com").get();
    			System.out.println(doc);
    			Element content = doc.getElementById("wrapper");//先观察要爬取的网站
    			System.out.println(content);
    			Elements links = content.getElementsByTag("a");
    			for (Element link : links) {
    				String linkHref = link.attr("href");
    				String linkText = link.text();
    				System.out.println(linkHref + "\t"+linkText);
    			}
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}

getElementById、getElementsByTag等方法都类似于jquery中的，太方便啦！

## 4、试用jsoup的选择器解析数据 ##

jsoup最大的方便之处就是它拥有匹配到CSS的选择器

try {
    			Document doc = Jsoup.connect("http://www.baidu.com").get();
    			System.out.println(doc);
    			Elements links = doc.select("a[href]"); // 具有 href 属性的链接
    			for(Element link:links){
    				System.out.println(link);
    			}
    			System.out.println("-----------------------------------");
    			Elements pngs = doc.select("img[src$=.png]");// 所有引用 png 图片的元素
    			for(Element png:pngs){
    				System.out.println(png);
    			}
    			System.out.println("-----------------------------------");
    			Elements resultLinks = doc.select("#cp > a"); //id为cp中的a
    			for(Element link:resultLinks){
    				System.out.println(link);
    			}
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}

选择器的基本用法：

<table style="margin-top:0px; margin-right:auto; margin-left:auto; padding:0px; vertical-align:baseline; font-size:16px; width:780px; clear:both; border-spacing:0px; font-family:Arial,sans-serif; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); color:rgb(50,50,50); margin-bottom:30px!important"> 
 <thead style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined"> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(90,90,90); font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif"> 
   <th colspan="1" style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; width:241px; border-bottom-width:1px!important; border-bottom-style:solid!important; border-bottom-color:rgb(255,255,255)!important; background-color:rgb(244,244,244)"> tagname</th> 
   <th colspan="1" style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; width:459px; border-bottom-width:1px!important; border-bottom-style:solid!important; border-bottom-color:rgb(255,255,255)!important; background-color:rgb(244,244,244)"> 使用标签名来定位，例如 a</th> 
  </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined"> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; background-color:inherit"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:0px; background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">ns|tag</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:0px; background-color:inherit"> 使用命名空间的标签定位，例如 fb:name 来查找 &lt;fb:name&gt; 元素</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">#id</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用元素 id 定位，例如 #logo</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224); background-color:inherit"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">.class</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用元素的 class 属性定位，例如 .head</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">[attribute]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用元素的属性进行定位，例如 [href] 表示检索具有 href 属性的所有元素</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224); background-color:inherit"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">[^attr]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用元素的属性名前缀进行定位，例如 [^data-] 用来查找 HTML5 的 dataset 属性</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">[attr=value]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用属性值进行定位，例如 [width=500] 定位所有 width 属性值为 500 的元素</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224); background-color:inherit"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">[attr^=value], [attr$=value], [attr*=value]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 这三个语法分别代表，属性以 value 开头、结尾以及包含</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">[attr~=regex]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 使用正则表达式进行属性值的过滤，例如 img[src~=(?i)\.(png|jpe?g)]</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224); background-color:inherit"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">*</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 定位所有元素</td> 
  </tr> 
 </tbody> 
</table>

组合用法：

<table style="margin-top:0px; margin-right:auto; margin-left:auto; padding:0px; vertical-align:baseline; font-size:16px; width:780px; clear:both; border-spacing:0px; font-family:Arial,sans-serif; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(221,221,221); color:rgb(50,50,50); margin-bottom:30px!important"> 
 <thead style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined"> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-bottom-width:1px; border-bottom-style:solid; border-bottom-color:rgb(90,90,90); font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif"> 
   <th style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-bottom-width:1px!important; border-bottom-style:solid!important; border-bottom-color:rgb(255,255,255)!important; background-color:rgb(244,244,244)"> el#id</th> 
   <th style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-bottom-width:1px!important; border-bottom-style:solid!important; border-bottom-color:rgb(255,255,255)!important; background-color:rgb(244,244,244)"> 定位 id 值某个元素，例如 a#logo -&gt; &lt;a id=logo href= … &gt;</th> 
  </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined"> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:0px; background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">el.class</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:0px; background-color:inherit"> 定位 class 为指定值的元素，例如 div.head -&gt; &lt;div class=head&gt;xxxx&lt;/div&gt;</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">el[attr]</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 定位所有定义了某属性的元素，例如 a[href]</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">以上三个任意组合</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 例如 a[href]#logo 、a[name].outerlink</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">ancestor child</span></td> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> 这五种都是元素之间组合关系的选择器语法，其中包括父子关系、合并关系和层次关系。</td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">parent &gt; child</span></td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">siblingA + siblingB</span></td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">siblingA ~ siblingX</span></td> 
  </tr> 
  <tr style="margin:0px; padding:0px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(224,224,224)"> 
   <td style="margin:0px; padding:16px 20px; vertical-align:baseline; font-size:undefined; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221); background-color:inherit"> <span style="margin:0px; padding:0px; border:0px; vertical-align:baseline; font-size:undefined; font-family:HelvBoldIBM,'Helvetica Neue',Arial,sans-serif">el, el, el</span></td> 
  </tr> 
 </tbody> 
</table>

[jsoup.jar]: http://download.csdn.net/detail/qq_29721419/9845802