网络爬虫之java HttpClient-Post

小鱼儿 2022-01-27 13:07 289阅读 0赞

## 利用HttpClient-Post去爬取数据 ##

#### 1、网络爬虫方法简述： ####

其实网络爬虫就跟我们人用浏览器去访问一个网页套路是一样的，都是分为四个部分：

1.  打开浏览器：创建HttpClient对象。
2.  输入网址：创建发起Post请求，创建HttpPost对象。
3.  按回车：发起请求，返回响应，使用httpClient发送请求。
4.  解析响应获取请求：判断状态码是否是200，如果为200就是访问成功了。
5.  最后关闭response和httpClient

#### 2、该项目的目录结构 ####

如何创建项目请看：[https://blog.csdn.net/weixin\_44588495/article/details/90580722][https_blog.csdn.net_weixin_44588495_article_details_90580722]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70]

#### 3、不带参数去爬取数据 ####

package com.crawler;
    
    import org.apache.http.HttpEntity;
    import org.apache.http.client.methods.CloseableHttpResponse;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.client.methods.HttpPost;
    import org.apache.http.impl.client.CloseableHttpClient;
    import org.apache.http.impl.client.HttpClients;
    import org.apache.http.util.EntityUtils;
    
    import java.io.IOException;
    
    public class HttpClientPostTest { 
        public static void main(String[] args) throws Exception { 
            //1、打开浏览器，创建HttpClient对象
            CloseableHttpClient httpClient = HttpClients.createDefault();
            //2、输入网址，创建发起Get请求，创建HttpGet对象
            HttpPost httpPost = new HttpPost("http://www.itcast.cn");
    
            CloseableHttpResponse response = null;
            try { 
                //3、按回车，发起请求，返回响应，使用httpClient发送请求
                response = httpClient.execute(httpPost);
                //4、解析响应获取请求，判断状态码是否是200
                if(response.getStatusLine().getStatusCode() == 200){ 
                    HttpEntity httpEntity = response.getEntity();
                    String content = EntityUtils.toString(httpEntity,"utf-8");
                    System.out.println(content.length());
                }
            } catch (IOException e) { 
                e.printStackTrace();
            }
            response.close();
            httpClient.close();
    
    
    
        }
    }

运行结果：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70 1]

#### 4、带参数去爬取数据 ####

由于Post请求像Get请求那样在地址栏中表示，所以需要用一个List去装参数。还需要创建一个表单对象，将参数放入表单对象中，然后再将表单对象添加到HttpPost对象中。

package com.crawler;
    
    import org.apache.http.HttpEntity;
    import org.apache.http.NameValuePair;
    import org.apache.http.client.entity.UrlEncodedFormEntity;
    import org.apache.http.client.methods.CloseableHttpResponse;
    import org.apache.http.client.methods.HttpPost;
    import org.apache.http.impl.client.CloseableHttpClient;
    import org.apache.http.impl.client.HttpClients;
    import org.apache.http.message.BasicNameValuePair;
    import org.apache.http.util.EntityUtils;
    
    import java.io.IOException;
    import java.util.ArrayList;
    import java.util.List;
    
    public class HttpClientPostParamTest { 
        public static void main(String[] args) throws Exception { 
            //1、打开浏览器，创建HttpClient对象
            CloseableHttpClient httpClient = HttpClients.createDefault();
            //2、输入网址，创建发起Get请求，创建HttpGet对象
            HttpPost httpPost = new HttpPost("http://yun.itheima.com/course");
            //3、声明List集合，封装表单中的参数
            List<NameValuePair> params = new ArrayList<NameValuePair>();
            //设置请求地址是：http://yun.itheima.com/course?keys=java
            params.add(new BasicNameValuePair("keys","java"));
            //创建表单的Emtity对象，第一个参数是封装好的表单数据，第二个参数是编码
            UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(params,"utf-8");
            //设置表单中的Emptity对象到Post请求中
            httpPost.setEntity(formEntity);
    
            CloseableHttpResponse response = null;
            try { 
                //3、按回车，发起请求，返回响应，使用httpClient发送请求
                response = httpClient.execute(httpPost);
                //4、解析响应获取请求，判断状态码是否是200
                if(response.getStatusLine().getStatusCode() == 200){ 
                    HttpEntity httpEntity = response.getEntity();
                    String content = EntityUtils.toString(httpEntity,"utf-8");
                    System.out.println(content.length());
                }
            } catch (IOException e) { 
                e.printStackTrace();
            }
            response.close();
            httpClient.close();
        }
    }

运行结果：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70 2]  
温馨提示：这次输出的爬虫数据与上一篇文章不同，输出的是爬取数据的长度。

[https_blog.csdn.net_weixin_44588495_article_details_90580722]: https://blog.csdn.net/weixin_44588495/article/details/90580722
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70]: /images/20220127/10819ad34c2448b2840c066bbb850fcf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70 1]: /images/20220127/7f45e3f5138e43f4aaee5c4bdcf975cb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDU4ODQ5NQ_size_16_color_FFFFFF_t_70 2]: /images/20220127/fb58f7d940a847f1b4dd4b0e69bc9648.png