简单介绍一些java爬虫框架

Bertha 。 2023-07-02 09:26 7阅读 0赞

# JsoupXpath #

### 官方文档:[http://jsoupxpath.wanghaomiao.cn/][http_jsoupxpath.wanghaomiao.cn] ###

### githup:[https://github.com/zhegexiaohuozi/JsoupXpath][https_github.com_zhegexiaohuozi_JsoupXpath] ###

## maven: ##

<dependency>
        <groupId>cn.wanghaomiao</groupId>
        <artifactId>JsoupXpath</artifactId>
        <version>2.3.2</version>
    </dependency>

## 示例: ##

String url = "https://www.qidian.com/rank/yuepiao?chn=21&page=1";
    Document document = Jsoup.connect(url).get();
    String xpath = ".//div[@class='book-mid-info']/h4/a/text()";
    JXDocument underTest = JXDocument.create(document.toString());
    List<Object> res = underTest.sel(xpath);
    for (Object re : res) {
      System.err.println(re);
    }

# slennium #

## 教程:[https://www.cnblogs.com/TankXiao/p/5252754.html][https_www.cnblogs.com_TankXiao_p_5252754.html] ##

##  ##

## maven ##

<dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>3.141.59</version>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-chrome-driver</artifactId>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-remote-driver</artifactId>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-api</artifactId>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-support</artifactId>
    </dependency>
    <dependency>
        <groupId>net.sourceforge.htmlcleaner</groupId>
        <artifactId>htmlcleaner</artifactId>
        <version>2.9</version>
    </dependency>

## 示例: ##

// 可省略，若驱动放在其他目录需指定驱动路径
    System.setProperty(
        "webdriver.chrome.driver",
        "D:\\Project\\java爬虫\\javapc01\\src\\test\\java\\com\\hw\\javapc\\chromedriver.exe");
    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.addArguments("--headless");
    ChromeDriver driver = new ChromeDriver(chromeOptions);
    driver.get("https://www.qidian.com/rank/yuepiao?chn=21&page=1");
    // 休眠1s,为了让js执行完
    Thread.sleep(1000L);
    // 网页源码
    String pageSource = driver.getPageSource();
    //使用xpath解析
    HtmlCleaner hc = new HtmlCleaner();
    TagNode tn = hc.clean(pageSource);
    Object[] objects = tn.evaluateXPath("//div[@class='book-mid-info']/h4/a/text()");
    for (Object object : objects) {
      System.out.println(object);
    }

# jsoup #

## 文档:[https://www.open-open.com/jsoup/ ][https_www.open-open.com_jsoup_][https://jsoup.org/apidocs/org/jsoup/nodes/Document.html][https_jsoup.org_apidocs_org_jsoup_nodes_Document.html] ##

## 案例:[https://www.cnblogs.com/sam-uncle/p/10922366.html][https_www.cnblogs.com_sam-uncle_p_10922366.html] ##

Document document = Jsoup.connect("https://www.qidian.com/rank/yuepiao?chn=21&page=1").get();
    Elements elementsByClass = document.getElementsByClass("book-mid-info");
    for (Element byClass : elementsByClass) {
    
      String text = byClass.select("h4 a[data-eid='qd_C40']").text();
      String user = byClass.select("p a[data-eid='qd_C41']").text();
      String type = byClass.select("p a[data-eid='qd_C42']").text();
      String trim = byClass.select("p[class='intro']").text().trim();
      log.info("小说名:{},作者:{},类型:{},介绍:{}",text,user,type,trim);
    }

# webmagic (有bug且maven仓库不更新) #

**类似python的scrapy.作者是参考scrapy写的,但是已经不怎么更新了**

## 文档:[http://webmagic.io/docs/zh/][http_webmagic.io_docs_zh] ##

## githup:[https://github.com/code4craft/webmagic][https_github.com_code4craft_webmagic] ##

## maven: ##

<dependency>
        <groupId>us.codecraft</groupId>
        <artifactId>webmagic-core</artifactId>
        <version>0.7.3</version>
    </dependency>
    <dependency>
        <groupId>us.codecraft</groupId>
        <artifactId>webmagic-extension</artifactId>
        <version>0.7.3</version>
    </dependency>

### java的生态还是非常不错的,什么类型的框架都有  python能弄的爬虫  java也一样可以的,  说到底语言不过是工具,真正用什么根据合适的情况考虑永不同的东西   简单爬虫就用jsoup和jsoupxpath就行了    ###

#  #

[http_jsoupxpath.wanghaomiao.cn]: http://jsoupxpath.wanghaomiao.cn/
[https_github.com_zhegexiaohuozi_JsoupXpath]: https://github.com/zhegexiaohuozi/JsoupXpath
[https_www.cnblogs.com_TankXiao_p_5252754.html]: https://www.cnblogs.com/TankXiao/p/5252754.html
[https_www.open-open.com_jsoup_]: https://www.open-open.com/jsoup/
[https_jsoup.org_apidocs_org_jsoup_nodes_Document.html]: https://jsoup.org/apidocs/org/jsoup/nodes/Document.html
[https_www.cnblogs.com_sam-uncle_p_10922366.html]: https://www.cnblogs.com/sam-uncle/p/10922366.html
[http_webmagic.io_docs_zh]: http://webmagic.io/docs/zh/
[https_github.com_code4craft_webmagic]: https://github.com/code4craft/webmagic