java爬虫-0010，资源篇

分手后的思念是犯贱 2022-05-09 07:52 156阅读 0赞

爬虫：模拟用户操作浏览器获取信息，即模拟发送request请求，获取response对象，解析获取自己需要的数据。

1、源码下载工具：

1.1 httpclient：可以抓取未经过js渲染的源代码，速度较快。

参考博客：[https://blog.csdn.net/lupangdelu/article/details/51007862][https_blog.csdn.net_lupangdelu_article_details_51007862]

[https://blog.csdn.net/chszs/article/details/16854747][https_blog.csdn.net_chszs_article_details_16854747]

1.2 phantomjs：可以抓取经过js渲染的源代码，速度慢。

参考博客：[https://www.cnblogs.com/front-Thinking/p/4321720.html][https_www.cnblogs.com_front-Thinking_p_4321720.html]

[http://javascript.ruanyifeng.com/tool/phantomjs.html][http_javascript.ruanyifeng.com_tool_phantomjs.html]

[https://blog.csdn.net/smile\_miracle/article/details/70817088][https_blog.csdn.net_smile_miracle_article_details_70817088]

2、源码清洗工具

2.1 jsoup

参考博客：[http://www.open-open.com/jsoup/][http_www.open-open.com_jsoup]

[http://www.open-open.com/jsoup/dom-navigation.htm][http_www.open-open.com_jsoup_dom-navigation.htm]

3、存储工具

3.1 berkeleydb：类似redis 内存数据库，速度快。

参考博客：[https://blog.csdn.net/jediael\_lu/article/details/27534223][https_blog.csdn.net_jediael_lu_article_details_27534223]

3.2mysql

4、抓包工具

4.1 chrome：开发者工具

参考博客：[https://blog.csdn.net/wrongyao/article/details/79877845][https_blog.csdn.net_wrongyao_article_details_79877845]

4.2 fiddler：可以设置代理抓取app中的数据包，获取路口

参考博客：[https://my.oschina.net/jhao104/blog/605963][https_my.oschina.net_jhao104_blog_605963]

5、反编译工具

参考博客：[https://blog.csdn.net/F0ED9cZN4Ly992G/article/details/78780254][https_blog.csdn.net_F0ED9cZN4Ly992G_article_details_78780254]

5.1 apktool 反编译可以查看资源文件等

5.2 dex2jar 可以将dex转化乘jar包

5.3 ja-gui ，luyten可以查看源码，结合使用

5.4 xposed 逆向神器，可以hook某个函数，得到参数等信息。

6、打包工具

6.1 exe4j

参考博客：[https://blog.csdn.net/qq\_20473985/article/details/53186216][https_blog.csdn.net_qq_20473985_article_details_53186216]

7、http协议

参考博客：[https://www.cnblogs.com/ranyonsue/p/5984001.html][https_www.cnblogs.com_ranyonsue_p_5984001.html]

[http://www.ruanyifeng.com/blog/2016/08/http.html][http_www.ruanyifeng.com_blog_2016_08_http.html]

[https_blog.csdn.net_lupangdelu_article_details_51007862]: https://blog.csdn.net/lupangdelu/article/details/51007862
[https_blog.csdn.net_chszs_article_details_16854747]: https://blog.csdn.net/chszs/article/details/16854747
[https_www.cnblogs.com_front-Thinking_p_4321720.html]: https://www.cnblogs.com/front-Thinking/p/4321720.html
[http_javascript.ruanyifeng.com_tool_phantomjs.html]: http://javascript.ruanyifeng.com/tool/phantomjs.html
[https_blog.csdn.net_smile_miracle_article_details_70817088]: https://blog.csdn.net/smile_miracle/article/details/70817088
[http_www.open-open.com_jsoup]: http://www.open-open.com/jsoup/
[http_www.open-open.com_jsoup_dom-navigation.htm]: http://www.open-open.com/jsoup/dom-navigation.htm
[https_blog.csdn.net_jediael_lu_article_details_27534223]: https://blog.csdn.net/jediael_lu/article/details/27534223
[https_blog.csdn.net_wrongyao_article_details_79877845]: https://blog.csdn.net/wrongyao/article/details/79877845
[https_my.oschina.net_jhao104_blog_605963]: https://my.oschina.net/jhao104/blog/605963
[https_blog.csdn.net_F0ED9cZN4Ly992G_article_details_78780254]: https://blog.csdn.net/F0ED9cZN4Ly992G/article/details/78780254
[https_blog.csdn.net_qq_20473985_article_details_53186216]: https://blog.csdn.net/qq_20473985/article/details/53186216
[https_www.cnblogs.com_ranyonsue_p_5984001.html]: https://www.cnblogs.com/ranyonsue/p/5984001.html
[http_www.ruanyifeng.com_blog_2016_08_http.html]: http://www.ruanyifeng.com/blog/2016/08/http.html