JSoup模拟登录新版正方教务系统（内网-教务系统）爬取信息过程详解

迷南。 2022-12-17 04:28 328阅读 0赞

新版正方教务系统登录界面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center]

### 目录 ###

*  一、需求分析
 *  二、模拟登录内网
 *  三、模拟登录教务系统
 *  四、爬取成绩和课表信息
 *  参考文章

# 一、需求分析 #

需要访问教务系统，爬取出课表成绩等信息，并在自己所写的APP上进行展示。由于访问教务系统需要连接校园网，所以本次爬取采用了“内网-教务系统”两级爬取策略，即先模拟登录校园内网，然后携带内网cookies登录教务系统，最终爬取相关信息。

# 二、模拟登录内网 #

内网登录界面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 1]  
URL：`https://webvpn.ncepu.edu.cn/users/sign_in`  
主要步骤：

1.  填好用户名以及登录密码，按下F12，并在Elements中搜索action：  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 2]  
    **可以看到，我们输入的表单数据最终被提交到了"/users/sign\_in"里。**
2.  点击登录，在Network里面找到sign\_in，可以看到我们模拟登录需要的各种信息：  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 3]
3.  开始写代码。

*  第一步主要是获取表单信息以及cookie，可能有的网站这里必须动态请求网页数据来获得form data，具体原因请参考：[Exception in thread “main“ org.jsoup.HttpStatusException: HTTP error fetching URL. Status=422, URL=][Exception in thread _main_ org.jsoup.HttpStatusException_ HTTP error fetching URL. Status_422_ URL]

Connection connection = Jsoup.connect(URL);
    connection.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36");
    Response res = connection.execute();   //获取res.cookies()，后面要用到
    Document d = Jsoup.parse(res.body());
    List<Element> elements = d.select("form");
    Map<String, String> datas = new HashMap<>();
    for (Element element : elements.get(0).getAllElements()) { 
        if (element.attr("name").equals("user[login]")) { 
            element.attr("value", "************");
        }
        if (element.attr("name").equals("user[password]")) { 
            e.attr("value", "******");
        }
        if (element.attr("name").length() > 0) { 
            datas.put(e.attr("name"), e.attr("value"));
        }
    }

USER\_AGENT等信息都在这里面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 4]  
我们可以打印一下datas：

> \{user\[dymatice\_code\]=unknown, utf8=?, commit=登录 Login, user\[login\]=马赛克, user\[password\]=马赛克, authenticity\_token=+BD3FgRXj+LsvgUpS81EKyU7SOF1B6eshSzfo3aMOSHD3LoMsx8ZP85vWNbm1PbPJGbgJqHVbFkTvHuSzDwI8A==\}

*  第二步就是提交表单信息以及cookies，进行模拟登录：

Connection connection2 = Jsoup.connect("https://webvpn.ncepu.edu.cn/users/sign_in");
    connection2.header(USER_AGENT, USER_AGENT_VALUE);
    Response response = connection2.ignoreContentType(true).followRedirects(true).method(Method.POST).data(datas).cookies(res.cookies()).execute();

*  最后一步：打印一下获得的html以及获得的cookies：

System.out.println(response.body());
    Map<String, String> map = response.cookies();
    for (String s : map.keySet()) { 
        System.out.println(s + " : " + map.get(s));
    }

# 三、模拟登录教务系统 #

我们模拟登录进入到了内网界面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 5]  
现在我们要模拟登录到新教务系统这个网页，进入到它的登录页面，也就是文章一开始给出的界面：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center]  
主要步骤如下：

1.  按照模拟登录校园内网的方式，查看需要提交哪些表单数据，这里就不再演示了，直接上代码：

//登录
    public boolean beginLogin() throws Exception{ 
    
        connection = Jsoup.connect(url+ "/jwglxt/xtgl/login_slogin.html").cookies(cookies_innet);
        connection.header("Content-Type","application/x-www-form-urlencoded;charset=utf-8");
        connection.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36");
    
        connection.data("csrftoken",csrftoken);
        connection.data("yhm",stuNum);
        connection.data("mm",password);
        connection.data("mm",password);
        connection.cookies(cookies).ignoreContentType(true)
                    .method(Connection.Method.POST).execute();
    
        response = connection.followRedirects(true).execute();
        document = Jsoup.parse(response.body());
            //登录成功
            //System.out.println(document);
        if(document.getElementById("tips") == null){ 
            System.out.println("欢迎登录");
            System.out.println(response.cookies());
            return true;
        }else{ 
            System.out.println(document.getElementById("tips").text());
            System.out.println(response.cookies());
            return false;
        }
    }

代码里面的cookies\_innet就是模拟登录内网获得的cookies。csrftoken需要额外获取，另外这里面的密码是加密了的，所以我们也需要获取对当前输入密码加密后的密码，代码如下：

// 获取csrftoken和Cookies，并没有出错
    private void getCsrftoken(){ 
        try{ 
            connection = Jsoup.connect(url+ "/jwglxt/xtgl/login_slogin.html?language=zh_CN&_t="+new Date().getTime()).cookies(cookies_innet);
            connection.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36");
            response = connection.followRedirects(true).execute();
            cookies = response.cookies();
            //保存csrftoken
            document = Jsoup.parse(response.body());
            csrftoken = document.getElementById("csrftoken").val();
        }catch (Exception ex){ 
            ex.printStackTrace();
        }
    }
        // 获取公钥并加密密码
    public void getRSApublickey() throws Exception{ 
        connection = Jsoup.connect(url+ "/jwglxt/xtgl/login_getPublicKey.html?" +
                    "time="+ new Date().getTime()).cookies(cookies_innet);
        connection.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36");
        response = connection.cookies(cookies).ignoreContentType(true).followRedirects(true).execute();
    
        JSONObject jsonObject = JSON.parseObject(response.body());
        modulus = jsonObject.getString("modulus");
        exponent = jsonObject.getString("exponent");
        password = RSAEncoder.RSAEncrypt(password, B64.b64tohex(modulus), B64.b64tohex(exponent));
        password = B64.hex2b64(password);
    }

附加的B64.java与RSAEncoder.java代码：

import static java.lang.Integer.parseInt;
    
    public class B64 { 
    
        public static String b64map="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
        private static char b64pad = '=';
        private static String hexCode = "0123456789abcdef";
    
        // 获取对应16进制字符
        public static char int2char(int a){ 
            return hexCode.charAt(a);
        }
    
        // Base64转16进制
        public static String b64tohex(String s) { 
            String ret = "";
            int k = 0;
            int slop = 0;
            for(int i = 0; i < s.length(); ++i) { 
                if(s.charAt(i) == b64pad) break;
                int v = b64map.indexOf(s.charAt(i));
                if(v < 0) continue;
                if(k == 0) { 
                    ret += int2char(v >> 2);
                    slop = v & 3;
                    k = 1;
                }
                else if(k == 1) { 
                    ret += int2char((slop << 2) | (v >> 4));
                    slop = v & 0xf;
                    k = 2;
                }
                else if(k == 2) { 
                    ret += int2char(slop);
                    ret += int2char(v >> 2);
                    slop = v & 3;
                    k = 3;
                }
                else { 
                    ret += int2char((slop << 2) | (v >> 4));
                    ret += int2char(v & 0xf);
                    k = 0;
                }
            }
            if(k == 1)
                ret += int2char(slop << 2);
            return ret;
        }
    
        // 16进制转Base64
        public static String hex2b64(String h) { 
            int i , c;
            StringBuilder ret = new StringBuilder();
            for(i = 0; i+3 <= h.length(); i+=3) { 
                c = parseInt(h.substring(i,i+3),16);
                ret.append(b64map.charAt(c >> 6));
                ret.append(b64map.charAt(c & 63));
            }
            if(i+1 == h.length()) { 
                c = parseInt(h.substring(i,i+1),16);
                ret.append(b64map.charAt(c << 2));
            }
            else if(i+2 == h.length()) { 
                c = parseInt(h.substring(i,i+2),16);
                ret.append(b64map.charAt(c >> 2));
                ret.append(b64map.charAt((c & 3) << 4));
            }
            while((ret.length() & 3) > 0) ret.append(b64pad);
            return ret.toString();
        }
    }

import java.math.BigInteger;
    import java.util.Random;
    
    public class RSAEncoder { 
        private static BigInteger n = null;
        private static BigInteger e = null;
    
        public static String RSAEncrypt(String pwd, String nStr, String eStr){ 
            n = new BigInteger(nStr,16);
            e = new BigInteger(eStr,16);
    
            BigInteger r = RSADoPublic(pkcs1pad2(pwd,(n.bitLength()+7)>>3));
            String sp = r.toString(16);
            if((sp.length()&1) != 0 )
                sp = "0" + sp;
            return sp;
        }
    
        private static BigInteger RSADoPublic(BigInteger x){ 
                  return x.modPow(e, n);
        }
    
        private static BigInteger pkcs1pad2(String s, int n){ 
            if(n < s.length() + 11) {  // TODO: fix for utf-8
                System.err.println("Message too long for RSAEncoder");
                return null;
            }
            byte[] ba = new byte[n];
            int i = s.length()-1;
            while(i >= 0 && n > 0) { 
                int c = s.codePointAt(i--);
                if(c < 128) {  // encode using utf-8
                    ba[--n] = new Byte(String.valueOf(c));
                }
                else if((c > 127) && (c < 2048)) { 
                     ba[--n] = new Byte(String.valueOf((c & 63) | 128));
                    ba[--n] = new Byte(String.valueOf((c >> 6) | 192));
                } else { 
                    ba[--n] = new Byte(String.valueOf((c & 63) | 128));
                    ba[--n] = new Byte(String.valueOf(((c >> 6) & 63) | 128));
                    ba[--n] = new Byte(String.valueOf((c >> 12) | 224));
                }
            }
            ba[--n] = new Byte("0");
    
            byte[] temp = new byte[1];
            Random rdm = new Random(47L);
    
            while(n > 2) {  // random non-zero pad
                temp[0] = new Byte("0");
                while(temp[0] == 0)
                    rdm.nextBytes(temp);
                ba[--n] = temp[0];
            }
            ba[--n] = 2;
            ba[--n] = 0;
            return new BigInteger(ba);
        }
    }

# 四、爬取成绩和课表信息 #

终于进入到了教务系统界面，接下来就是爬取成绩和课表信息，然后在自己写的APP中进行展示，效果如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 6]  
不过我们还是得一步步来：

1.  获取成绩信息。与前面类似，也需要提交表单数据，过程一模一样，需要提交哪些数据可以参照这篇博文：[爬虫时怎么查看需要提交哪些表单数据？][Link 1]  
    这里直接上代码：

// 获取成绩信息
    public void getStudentGrade(int year , int term) throws Exception { 
        Map<String,String> datas = new HashMap<>();
        datas.put("xnm",String.valueOf(year));
        datas.put("xqm",String.valueOf(term * term * 3));
        datas.put("_search","false");
        datas.put("nd",String.valueOf(new Date().getTime()));
        datas.put("queryModel.showCount","80");
        datas.put("queryModel.currentPage","1");
        datas.put("queryModel.sortName","");
        datas.put("queryModel.sortOrder","asc");
        datas.put("queryModel.sortName","");
        datas.put("time","0");
    
        System.out.println(datas);
    
        connection = Jsoup.connect(url+ "/jwglxt/cjcx/cjcx_cxDgXscj.html?gnmkdm=N305005&layout=default&su=" + stuNum);
        connection.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
        response = connection.cookies(cookies_innet).cookies(cookies).method(Connection.Method.POST)
                    .data(datas).ignoreContentType(true).execute();
        connection = Jsoup.connect(url+ "/jwglxt/cjcx/cjcx_cxDgXscj.html?doType=query&gnmkdm=N305005");
        connection.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
        response = connection.cookies(cookies_innet).cookies(cookies).method(Connection.Method.POST)
                    .data(datas).ignoreContentType(true).execute();
        System.out.println(response.body());
        JSONObject jsonObject = JSON.parseObject(response.body());
    
    
            //System.out.println(jsonObject);
        JSONArray gradeTable = JSON.parseArray(jsonObject.getString("items"));
            //System.out.println(gradeTable);
        for (Iterator iterator = gradeTable.iterator(); iterator.hasNext();) { 
            JSONObject lesson = (JSONObject) iterator.next();
            System.out.println(lesson.getString("kcmc") + " " +
                    lesson.getString("jsxm") + " " +
                    lesson.getString("bfzcj") + " " +
                    lesson.getString("jd") + " " +
                    lesson.getString("kcxzmc"));
        }
    }

有一点需要注意：提交参数中的showCount最好大一点，因为我们默认只爬取了第一页的数据，在第一页显示所有成绩信息才能一次性爬取完。

1.  成绩信息展示

[安卓ExpandableListView的详细使用教程（附代码解析过程）][ExpandableListView]

# 参考文章 #

这些文章都是我自己写的，算是对前面零散知识点的一点总结吧：

1.  [JSoup模拟登录网站（以校园内网为例）][JSoup]
2.  [JSoup利用获得的cookies访问该网页中的其它链接][JSoup_cookies]
3.  [爬虫时怎么查看需要提交哪些表单数据？][Link 1]
4.  [JSoup携带cookies连续跳转登录多个界面][JSoup_cookies 1]
5.  [Java爬虫简单判断是否模拟登录成功（以JSoup为例）][Java_JSoup]
6.  [安卓ExpandableListView的详细使用教程（附代码解析过程）][ExpandableListView]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/8f657636b6a440bdb7249b0acbf1174e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221123/7689a1eea1734b44a21d8895cac0bc90.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221123/8e18baec4f32475f8193cd1e60c1a853.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221123/22925341a3fb440380c683985d9e3a0d.png
[Exception in thread _main_ org.jsoup.HttpStatusException_ HTTP error fetching URL. Status_422_ URL]: https://blog.csdn.net/Cyril_KI/article/details/108172277
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221123/95ad9039079c45028a4157b8ab9efaf0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20221123/c3013434db124e1daaedff21f92297e8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0N5cmlsX0tJ_size_16_color_FFFFFF_t_70_pic_center 6]: /images/20221123/ab284cc7b73a4f138ef5027c0d2260c1.png
[Link 1]: https://blog.csdn.net/Cyril_KI/article/details/108300035
[ExpandableListView]: https://blog.csdn.net/Cyril_KI/article/details/108273802
[JSoup]: https://blog.csdn.net/Cyril_KI/article/details/108172515
[JSoup_cookies]: https://blog.csdn.net/Cyril_KI/article/details/108173210
[JSoup_cookies 1]: https://blog.csdn.net/Cyril_KI/article/details/108175514
[Java_JSoup]: https://blog.csdn.net/Cyril_KI/article/details/108440697