Jsoup爬虫之PhanTomJs工具类

ゝ一纸荒年。 2022-03-18 12:48 377阅读 0赞

用java的一款HTML解析器Jsoup爬取网页数据，有时候会碰到信息浏览器中有显示，但无法抓取信息的情况，这是因为该数据是由javascript元素加载完成事件而获取到的，这时候我们可以使用PhanTomJs工具模拟浏览器运行;  
PhanTomJs官网下载：[http://phantomjs.org/download.html][http_phantomjs.org_download.html]  
话不多说，为大家带来该工具的依赖jar包，配置文件及工具类，可通过工具类直接获得document对象  
配置PhanTomJs依赖jar包下载地址：  
配置文件:[https://pan.baidu.com/s/1BWXRyyId0bDtoSigxnpcnA][https_pan.baidu.com_s_1BWXRyyId0bDtoSigxnpcnA]  
phtantonjs.properties

javascriptEnabled=true
    takeScreenshot=true
    handlesAlerts=false
    databaseEnabled=false
    locationContextEnabled=false
    applicationCacheEnabled=false
    browserConnectionEnabled"=false
    cssSelectorsEnabled=true
    webStorageEnabled=false
    rotatable=false
    acceptSslCerts=false
    nativeEvents=true
    property=f://workApp//phantomjs-2.1.1-windows//bin//phantomjs.exe

工具类  
JsoupUtils

import org.jsoup.Jsoup;
    import org.jsoup.nodes.Document;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriverService;
    import org.openqa.selenium.remote.DesiredCapabilities;
    
    import java.util.ResourceBundle;
    import java.util.concurrent.TimeUnit;
    
    public class JsoupUtils {
        private static DesiredCapabilities dcaps = new DesiredCapabilities();
        private static ResourceBundle ps = ResourceBundle.getBundle("phantomjs");
        private static WebDriver driver =null;
        private static Document document=null;
        static {
            dcaps.setCapability("acceptSslCerts",Boolean.parseBoolean(ps.getString("acceptSslCerts")));
            dcaps.setCapability("takeScreenshot",Boolean.parseBoolean(ps.getString("takeScreenshot")));
            dcaps.setCapability("cssSelectorsEnabled",Boolean.parseBoolean(ps.getString("cssSelectorsEnabled")));
            dcaps.setJavascriptEnabled(Boolean.parseBoolean(ps.getString("javascriptEnabled")));
            dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,ps.getString("property"));
        }
        public static WebDriver getDriver(){
            return new PhantomJSDriver(dcaps);
        }
        public static Document getDocument(String src, int time,long sleep){
            driver=getDriver();
            try {
                driver.manage().timeouts().implicitlyWait(time, TimeUnit.SECONDS);
                driver.get(src);
                Thread.sleep(sleep);
            } catch (InterruptedException e) {
                System.out.println("被打断错误");
            }
            String html = driver.getPageSource();
            return document=Jsoup.parse(html);
        }
        public static void cloose(){
            driver.close();
            driver.quit();
        }
    }

测试代码

public class GrepJDProductDataTest2 {
        public static void main(String[] args) {
            try {
                Document document = JsoupUtils.getDocument("https://item.jd.com/100000220633.html", 1, 1001);
                System.out.println(document.select("[class='price']"));
            } catch (Exception e) {
                e.printStackTrace();
            }finally {
                JsoupUtils.cloose();
            }
        }
    }

测试效果图  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU5NDA1Ng_size_16_color_FFFFFF_t_70]  
第一次写文章如对各位朋友有帮助希望可以点个赞支持一下，谢谢！

[http_phantomjs.org_download.html]: http://phantomjs.org/download.html
[https_pan.baidu.com_s_1BWXRyyId0bDtoSigxnpcnA]: https://pan.baidu.com/s/1BWXRyyId0bDtoSigxnpcnA
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU5NDA1Ng_size_16_color_FFFFFF_t_70]: /images/20220318/c2ed33557e6f46b383cb3b2758a033b7.png