爬虫如何实现每天爬取，定点爬取[以股票数据为例]

矫情吗；* 2022-07-11 08:40 610阅读 0赞

*  分析抓取的数据
 *  抓包
 *  框架
 *  model
 *  main
 *  util
 *  parse
 *  db
 *  问题所在
 *  解决方法
    
     *  job
     *  jobmain

**近期，有人将本人博客，复制下来，直接上传到百度文库等平台。**   
**本文为原创博客，仅供技术学习使用。未经允许，禁止将其复制下来上传到百度文库等平台。如有转载请注明本文博客的地址(链接)**

# 分析抓取的数据 #

本文是以东方财富网的数据为例，**这里只做技术学习使用，请勿滥用**。如本文要抓取的数据是东方财富网的汽车板块及石油板块数据。如下为其地址：[http://quote.eastmoney.com/center/list.html\#28002481\_0\_2][http_quote.eastmoney.com_center_list.html_28002481_0_2]  
[http://quote.eastmoney.com/center/list.html\#28002464\_0\_2][http_quote.eastmoney.com_center_list.html_28002464_0_2]  
如下截图为其数据格式。

![这里写图片描述][SouthEast]

# 抓包 #

**写爬虫第一步是做网络抓包，这个我之前的博客中已经讲到**。**即看数据请求的真实地址**。关于本文为什么这样设计，请看我的专题博客，爬虫原理及相关基础：[http://blog.csdn.net/column/details/14269.html][http_blog.csdn.net_column_details_14269.html]。  
![这里写图片描述][SouthEast 1]

从上图中，可以看出数据真实的请求地址及请求的方法。而获得的是json数组。如下图所示：  
![这里写图片描述][SouthEast 2]

# 框架 #

本文使用的框架，如下图所示：  
![这里写图片描述][SouthEast 3]

**db:**主要放的是数据库操作文件，包含MyDataSource【数据库驱动注册、连接数据库的用户名、密码】，MYSQLControl【连接数据库，插入操作、更新操作、建表操作等】。

**model：**用来封装对象，说的直白一些，封装的就是我要操作数据对应的属性名。有不明白的看之前写的一个简单的网络爬虫([http://blog.csdn.net/qy20115549/article/details/52203722][http_blog.csdn.net_qy20115549_article_details_52203722])。

**parse：**这里面存放的是针对util获取的文件，进行解析，一般采用Jsoup解析；若是针对json数据，可采用正则表达式或者fastjson工具进行解析，建议使用fastjson，因其操作简单，快捷。

**main：**程序起点，也是重点，获取数据，执行数据库语句，存放数据。

**job：**用来执行的job任务。

**jobmain：**控制器，即合适执行job，如本文中的每天执行一次job。股票数据每天下午3点钟收盘，即设置为3点钟以后的某个时间点开始爬行相关股票数据。

# model #

model用来封装我要爬去的数据，如当天的日期，股票的id，股票的名称，股票价格等等。如下面程序：

package model;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public class ExtMarketOilStockModel { 
        private String date;
        private String stock_id;
        private String stock_name;
        private float stock_price;
        private float stock_change;
        private float stock_range;
        private float stock_amplitude;
        private int stock_trading_number;
        private int stock_trading_value;
        private float stock_yesterdayfinish_price;
        private float stock_todaystart_price;
        private float stock_max_price;
        private float stock_min_price;
        private float stock_fiveminuate_change;
        private String craw_time;
        public String getDate() {
            return date;
        }
        public void setDate(String date) {
            this.date = date;
        }
    
        public String getStock_id() {
            return stock_id;
        }
        public void setStock_id(String stock_id) {
            this.stock_id = stock_id;
        }
        public String getStock_name() {
            return stock_name;
        }
        public void setStock_name(String stock_name) {
            this.stock_name = stock_name;
        }
        public float getStock_price() {
            return stock_price;
        }
        public void setStock_price(float stock_price) {
            this.stock_price = stock_price;
        }
        public float getStock_change() {
            return stock_change;
        }
        public void setStock_change(float stock_change) {
            this.stock_change = stock_change;
        }
        public float getStock_range() {
            return stock_range;
        }
        public void setStock_range(float stock_range) {
            this.stock_range = stock_range;
        }
        public float getStock_amplitude() {
            return stock_amplitude;
        }
        public void setStock_amplitude(float stock_amplitude) {
            this.stock_amplitude = stock_amplitude;
        }
    
        public int getStock_trading_number() {
            return stock_trading_number;
        }
        public void setStock_trading_number(int stock_trading_number) {
            this.stock_trading_number = stock_trading_number;
        }
        public int getStock_trading_value() {
            return stock_trading_value;
        }
        public void setStock_trading_value(int stock_trading_value) {
            this.stock_trading_value = stock_trading_value;
        }
        public float getStock_yesterdayfinish_price() {
            return stock_yesterdayfinish_price;
        }
        public void setStock_yesterdayfinish_price(float stock_yesterdayfinish_price) {
            this.stock_yesterdayfinish_price = stock_yesterdayfinish_price;
        }
        public float getStock_todaystart_price() {
            return stock_todaystart_price;
        }
        public void setStock_todaystart_price(float stock_todaystart_price) {
            this.stock_todaystart_price = stock_todaystart_price;
        }
        public float getStock_max_price() {
            return stock_max_price;
        }
        public void setStock_max_price(float stock_max_price) {
            this.stock_max_price = stock_max_price;
        }
        public float getStock_min_price() {
            return stock_min_price;
        }
        public void setStock_min_price(float stock_min_price) {
            this.stock_min_price = stock_min_price;
        }
        public float getStock_fiveminuate_change() {
            return stock_fiveminuate_change;
        }
        public void setStock_fiveminuate_change(float stock_fiveminuate_change) {
            this.stock_fiveminuate_change = stock_fiveminuate_change;
        }
        public String getCraw_time() {
            return craw_time;
        }
        public void setCraw_time(String craw_time) {
            this.craw_time = craw_time;
        }
    }

# main #

主方法，尽量要求简单，这里我就这样写了。这里面有注释，很好理解。

package navi.main;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    import java.util.ArrayList;
    import java.util.List;
    
    import db.MYSQLControl;
    import model.ExtMarketOilStockModel;
    import parse.ExtMarketOilStockParse;
    
    public class ExtMarketOilStockMain { 
    
        public static void main(String[] args) throws Exception {
            List<String> urloillist=new ArrayList<String>();
            List<String> urlcarlist=new ArrayList<String>();
            List<ExtMarketOilStockModel> oilstocks=new ArrayList<ExtMarketOilStockModel>();
            List<ExtMarketOilStockModel> carstocks=new ArrayList<ExtMarketOilStockModel>();
            //石油相关股票就两页，对应两个地址
            String url1="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=1&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.13204790262127375";
            String url2="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=2&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.6972178580603532";
            urloillist.add(url1);
            urloillist.add(url2);
            for (int i = 0; i < urloillist.size(); i++) {
                //解析url
                oilstocks=ExtMarketOilStockParse.parseurl(urloillist.get(i));
                //存储每页的数据
                MYSQLControl.insertoilStocks(oilstocks);
            }
            //汽车相关股票有6页，对应6个地址
            for (int i = 1; i <6; i++) {
                String urli="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04811&sty=FCOIATA&sortType=C&sortRule=-1&page="+i+"&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.23492960370783944";
                urlcarlist.add(urli);
            }
            for (int i = 0; i < urlcarlist.size(); i++) {
                //解析url
                carstocks=ExtMarketOilStockParse.parseurl(urlcarlist.get(i));
                //存储数据
                MYSQLControl.insertcarStocks(carstocks);
            }
    
        }
    
    }

# util #

这里有三个文件，HTTPUtils，TimeUtils(这是我自己经常用的一个类，主要是各种日期的转化，如String转化为date，获取当前时间等等)，UumericalUtil(这是一个Float保留几位小数的类)。

package util;
    import java.io.BufferedReader;
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.InputStreamReader;
    import java.net.URL;
    import java.net.URLConnection;
    
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public abstract class HTTPUtils { 
        //这个方法是向后台请求数据，获取html或者json等
        public static String  getRawHtml(String personalUrl) throws InterruptedException,IOException {
            URL url = new URL(personalUrl);
            URLConnection conn = url.openConnection();
            InputStream in=null;
            try {
                conn.setConnectTimeout(3000);
                in = conn.getInputStream();
            } catch (Exception e) {
            }
            //将获取的数据转化为String
            String html = convertStreamToString(in);
            return html;
        }
        //这个方法是将InputStream转化为String
        public static String convertStreamToString(InputStream is) throws IOException {
            if (is == null)
                return "";
            BufferedReader reader = new BufferedReader(new InputStreamReader(is,"utf-8"));
            StringBuilder sb = new StringBuilder();
            String line = null;
            try {
                while ((line = reader.readLine()) != null) {
                    sb.append(line);
                }
            } catch (IOException e) {
                e.printStackTrace();
            } finally {
                try {
                    is.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            reader.close();
            return sb.toString();
    
        }
    }

**以下类是用来处理各种时间格式之间的转化，大家以后也可以使用。**

package util;
    
    import java.text.DateFormat;
    import java.text.DecimalFormat;
    import java.text.ParseException;
    import java.text.SimpleDateFormat;
    import java.util.ArrayList;
    import java.util.Calendar;
    import java.util.Date;
    import java.util.List;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public class TimeUtils { 
    
        public static void main( String[] args ) throws ParseException{
    
            String time = getMonth("2002-1-08 14:50:38");
            System.out.println(time);
            System.out.println(getDay("2002-1-08 14:50:38"));
            System.out.println(TimeUtils.parseTime("2016-05-19 19:17","yyyy-MM-dd HH:mm"));
    
        }
        //get current time
        public static String GetNowDate(String formate){  
            String temp_str="";  
            Date dt = new Date();  
            SimpleDateFormat sdf = new SimpleDateFormat(formate);  
            temp_str=sdf.format(dt);  
            return temp_str;  
        }  
        public static String getMonth( String time ){
    
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM");
            Date date = null;
            try {
    
                date = sdf.parse(time);
                Calendar cal = Calendar.getInstance();
                cal.setTime(date);
    
            } catch (ParseException e) {
                e.printStackTrace();
            }
    
            return sdf.format(date);
    
        }
    
        public static String getDay( String time ){
    
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
            Date date = null;
            try {
    
                date = sdf.parse(time);
                Calendar cal = Calendar.getInstance();
                cal.setTime(date);
    
            } catch (ParseException e) {
                e.printStackTrace();
            }
    
            return sdf.format(date);
    
        }
    
        public static Date parseTime(String inputTime) throws ParseException{
    
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");  
            Date date = sdf.parse(inputTime); 
    
            return date;
    
        }
        public static String dateToString(Date date, String type) { 
            DateFormat df = new SimpleDateFormat(type);  
            return df.format(date);  
        }
        public static Date parseTime(String inputTime, String timeFormat) throws ParseException{
    
            SimpleDateFormat sdf = new SimpleDateFormat(timeFormat);  
            Date date = sdf.parse(inputTime); 
    
            return date;
    
        }
    
        public static Calendar parseTimeToCal(String inputTime, String timeFormat) throws ParseException{
    
            SimpleDateFormat sdf = new SimpleDateFormat(timeFormat);  
            Date date = sdf.parse(inputTime); 
            Calendar calendar = Calendar.getInstance();
            calendar.setTime(date);
    
            return calendar;
    
        }
    
        public static int getDaysBetweenCals(Calendar cal1, Calendar cal2) throws ParseException{
    
            return (int) ((cal2.getTimeInMillis()-cal1.getTimeInMillis())/(1000*24*3600));
    
        }
    
        public static Date parseTime(long inputTime){
    
            // SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
            Date date= new Date(inputTime);
            return date;
    
        }
    
        public static String parseTimeString(long inputTime){
    
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
            Date date= new Date(inputTime);
            return sdf.format(date);
    
        }
        public static String parseStringTime(String inputTime){
    
            String date=null;
            try {
                Date date1 = new SimpleDateFormat("yyyyMMddHHmmss").parse(inputTime);
                date=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(date1);
            } catch (ParseException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
    
            return date;
        }
        public static List<String> YearMonth(int year) {
            List<String> yearmouthlist=new ArrayList<String>();
            for (int i = 1; i < 13; i++) {
                DecimalFormat dfInt=new DecimalFormat("00");
                String sInt = dfInt.format(i);
                yearmouthlist.add(year+sInt);
            }
    
            return yearmouthlist;
        } 
        public static List<String> YearMonth(int startyear,int finistyear) {
            List<String> yearmouthlist=new ArrayList<String>();
            for (int i = startyear; i < finistyear+1; i++) {
                for (int j = 1; j < 13; j++) {
                    DecimalFormat dfInt=new DecimalFormat("00");
                    String sInt = dfInt.format(j);
                    yearmouthlist.add(i +"-"+sInt);
                }
            }
            return yearmouthlist;
        } 
        public static List<String> TOAllDay(int year){
            List<String> daylist=new ArrayList<String>();
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); 
            int m=1;//月份计数 
            while (m<13) 
            { 
                int month=m; 
                Calendar cal=Calendar.getInstance();//获得当前日期对象 
                cal.clear();//清除信息 
                cal.set(Calendar.YEAR,year); 
                cal.set(Calendar.MONTH,month-1);//1月从0开始 
                cal.set(Calendar.DAY_OF_MONTH,1);//设置为1号,当前日期既为本月第一天 
    
                System.out.println("##########___" + sdf.format(cal.getTime())); 
    
                int count=cal.getActualMaximum(Calendar.DAY_OF_MONTH); 
    
                System.out.println("$$$$$$$$$$________" + count); 
    
                for (int j=0;j<=(count - 2);) 
                { 
                    cal.add(Calendar.DAY_OF_MONTH,+1); 
                    j++; 
                    daylist.add(sdf.format(cal.getTime()));
                } 
                m++; 
            } 
            return daylist;
        }
        //获取昨天的日期
        public static String getyesterday(){
            Calendar   cal   =   Calendar.getInstance();
            cal.add(Calendar.DATE,   -1);
            String yesterday = new SimpleDateFormat( "yyyy-MM-dd ").format(cal.getTime());
            return yesterday;
        }
    }

这个类实现的是保留几位小数。如股票价格等，保留两位小数。

package util;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    import java.math.BigDecimal;
    import java.text.DecimalFormat;
    
    public class UumericalUtil { 
    
        public static float FloatTO(float f, int number) {
            BigDecimal   b  =   new BigDecimal(f);  
            float   f1   =  b.setScale(number, BigDecimal.ROUND_HALF_UP).floatValue();  
            return f1;  
        }  
        public static String NumberTO(int number) {
            DecimalFormat dfInt=new DecimalFormat("00");
            String sInt = dfInt.format(number);
            System.out.println(sInt);
            return sInt;
        } 
    
    }

# parse #

parse主要是通过Jsoup或者其他工具来解析html文件。并将解析后的数据，封装在List集合中，将数据通过层层返回到main方法中。如这里只是采用最简单的字符串解析的方式。如下为某一页的数据，这要针对的是此类型的数据进行解析：

var quote_123={rank:["2,002662,京威股份,15.62,0.38,2.49%,2.95,10294,15948185,15.24,15.28,15.65,15.20,-,-,-,-,-,-,-,-,0.00%,0.62,0.17,33.47","2,002536,西泵股份,13.15,0.32,2.49%,3.74,26558,34710121,12.83,12.88,13.27,12.79,-,-,-,-,-,-,-,-,0.00%,0.99,0.87,41.09","1,600741,华域汽车,16.22,0.39,2.46%,2.59,215140,346480560,15.83,15.85,16.26,15.85,-,-,-,-,-,-,-,-,0.12%,1.23,0.75,8.59","1,601689,拓普集团,29.74,0.68,2.34%,3.20,36329,107964394,29.06,29.06,29.94,29.01,-,-,-,-,-,-,-,-,-0.20%,1.34,2.13,34.32","1,603306,华懋科技,33.87,0.74,2.23%,4.50,9251,31242113,33.13,33.14,34.20,32.71,-,-,-,-,-,-,-,-,-0.03%,0.72,1.25,29.60","1,601799,星宇股份,37.40,0.80,2.19%,3.80,5522,20477010,36.60,36.40,37.50,36.11,-,-,-,-,-,-,-,-,0.03%,0.86,0.23,28.43","1,603166,福达股份,14.02,0.29,2.11%,2.91,47265,66170428,13.73,13.80,14.14,13.74,-,-,-,-,-,-,-,-,0.21%,0.96,3.15,95.59","2,002190,成飞集成,32.44,0.66,2.08%,2.99,25213,81219488,31.78,31.63,32.58,31.63,-,-,-,-,-,-,-,-,0.03%,0.86,0.73,93.58","1,600213,亚星客车,14.77,0.30,2.07%,3.46,18878,27820060,14.47,14.52,14.88,14.38,-,-,-,-,-,-,-,-,-0.07%,0.64,0.86,55.39","2,300432,富临精工,21.28,0.43,2.06%,4.70,28707,60945368,20.85,20.60,21.58,20.60,-,-,-,-,-,-,-,-,-0.14%,1.29,2.07,50.58","2,300375,鹏翎股份,21.25,0.42,2.02%,3.94,11367,24164157,20.83,20.83,21.45,20.63,-,-,-,-,-,-,-,-,-0.14%,0.83,1.44,30.27","2,002363,隆基机械,11.47,0.22,1.96%,2.49,33946,38796837,11.25,11.27,11.55,11.27,-,-,-,-,-,-,-,-,0.00%,0.80,0.88,61.45","1,600469,风神股份,11.55,0.22,1.94%,3.09,38444,44305565,11.33,11.33,11.63,11.28,-,-,-,-,-,-,-,-,0.09%,0.67,0.68,27.07","2,002454,松芝股份,12.98,0.24,1.88%,2.83,27839,36056020,12.74,12.70,13.06,12.70,-,-,-,-,-,-,-,-,0.00%,1.17,0.87,25.84","2,002488,金固股份,14.79,0.27,1.86%,2.48,29002,42872475,14.52,14.52,14.88,14.52,-,-,-,-,-,-,-,-,0.00%,0.72,0.75,-","2,002284,亚太股份,13.18,0.24,1.85%,3.32,61756,81198133,12.94,12.87,13.30,12.87,-,-,-,-,-,-,-,-,0.30%,1.10,0.90,58.15","1,603788,宁波高发,35.97,0.64,1.81%,3.40,6719,24160418,35.33,35.21,36.33,35.13,-,-,-,-,-,-,-,-,0.03%,0.59,1.37,34.10","2,000957,中通客车,14.36,0.25,1.77%,2.69,59696,85581415,14.11,14.07,14.45,14.07,-,-,-,-,-,-,-,-,0.00%,0.79,1.25,13.99","2,300304,云意电气,52.12,0.90,1.76%,5.70,179330,922614032,51.22,50.38,52.83,49.91,-,-,-,-,-,-,-,-,-0.04%,1.12,9.35,108.58","2,002607,亚夏汽车,10.03,0.17,1.72%,4.16,27760,27878904,9.86,9.89,10.19,9.78,-,-,-,-,-,-,-,-,-0.30%,0.97,1.03,57.87"],pages:6}

package parse;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    import java.util.ArrayList;
    import java.util.List;
    import model.ExtMarketOilStockModel;
    import util.HTTPUtils;
    import util.TimeUtils;
    import util.UumericalUtil;
    public class ExtMarketOilStockParse {
        public static List<ExtMarketOilStockModel> parseurl(String url) throws Exception {
            List<ExtMarketOilStockModel> list=new ArrayList<ExtMarketOilStockModel>();
            String response=HTTPUtils.getRawHtml(url);
            String html = response.toString();
            String jsonarra=html.split("rank:")[1].split(",pages")[0];
            String stocks[]=jsonarra.split("\",");
            List<String> stocklist=new ArrayList<String>();
            for (int i = 0; i < stocks.length; i++) { 
                stocklist.add(stocks[i].replace("[\"", "").replace("\"", "").replace("]", ""));
                System.out.println(stocks[i].replace("[\"", "").replace("\"", "").replace("]", ""));
            }
            for (int i = 0; i < stocklist.size(); i++) { 
                String date=TimeUtils.GetNowDate("yyyy-MM-dd");
                String stock_id=stocklist.get(i).split(",")[1];
                String stock_name=stocklist.get(i).split(",")[2];
                float stock_price=0;
                float stock_change=0;
                float stock_range=0;
                float stock_amplitude=0;
                int stock_trading_number=0;
                int stock_trading_value=0;
                float stock_yesterdayfinish_price=0;
                float stock_todaystart_price=0;
                float stock_max_price=0;
                float stock_min_price=0;
                float stock_fiveminuate_change=0;
                if (!stocklist.get(i).split(",")[3].equals("-")) {
                    //价格
                    stock_price=Float.parseFloat(stocklist.get(i).split(",")[3]);
                    //涨跌额
                    stock_change=Float.parseFloat(stocklist.get(i).split(",")[4]);
                    System.out.println(stock_change);
                    //涨跌幅
                    stock_range=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[5].replace("%", ""))*0.01),4);
                    stock_amplitude=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[6].replace("%", ""))*0.01),4);;
                    stock_trading_number=Integer.parseInt(stocklist.get(i).split(",")[7].replace("%", ""));
                    stock_trading_value=Integer.parseInt(stocklist.get(i).split(",")[8].replace("%", ""));
                    stock_yesterdayfinish_price=Float.parseFloat(stocklist.get(i).split(",")[9]);
                    stock_todaystart_price=Float.parseFloat(stocklist.get(i).split(",")[10]);
                    stock_max_price=Float.parseFloat(stocklist.get(i).split(",")[11]);
                    stock_min_price=Float.parseFloat(stocklist.get(i).split(",")[12]);
                    stock_fiveminuate_change=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[21].replace("%", ""))*0.01),4);;
                    System.out.println(stock_fiveminuate_change);
                }
                String craw_time=TimeUtils.GetNowDate("yyyy-MM-dd HH:mm:ss");
                ExtMarketOilStockModel model=new ExtMarketOilStockModel();
                model.setDate(date);
                model.setStock_id(stock_id);
                model.setStock_name(stock_name);
                model.setStock_price(stock_price);
                model.setStock_change(stock_change);
                model.setStock_range(stock_range);
                model.setStock_amplitude(stock_amplitude);
                model.setStock_trading_number(stock_trading_number);
                model.setStock_trading_value(stock_trading_value);
                model.setStock_yesterdayfinish_price(stock_yesterdayfinish_price);
                model.setStock_todaystart_price(stock_todaystart_price);
                model.setStock_max_price(stock_max_price);
                model.setStock_min_price(stock_min_price);
                model.setStock_fiveminuate_change(stock_fiveminuate_change);
                model.setCraw_time(craw_time);
                list.add(model);
            }
            return list; 
        }
    }

# db #

db中包含两个java文件，MyDataSource，MYSQLControl。这两个文件的作用已在前面说明了。

package db;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    import javax.sql.DataSource;
    import org.apache.commons.dbcp2.BasicDataSource;
    
    public class MyDataSource { 
    
        public static DataSource getDataSource(String connectURI){
    
            BasicDataSource ds = new BasicDataSource();
             //MySQL的jdbc驱动
            ds.setDriverClassName("com.mysql.jdbc.Driver");
            ds.setUsername("root");              //所要连接的数据库名
            ds.setPassword("112233");                //MySQL的登陆密码
            ds.setUrl(connectURI);
    
            return ds;
    
        }
    
    }

package db;
    import java.sql.SQLException;
    import java.util.List;
    import javax.sql.DataSource;
    import org.apache.commons.dbutils.QueryRunner;
    import org.apache.commons.dbutils.ResultSetHandler;
    import org.apache.commons.dbutils.handlers.BeanListHandler;
    import org.apache.commons.dbutils.handlers.ColumnListHandler;
    import org.apache.commons.dbutils.handlers.ScalarHandler;
    import org.apache.commons.logging.Log;
    import org.apache.commons.logging.LogFactory;
    import model.ExtMarketOilStockModel;
    /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public class MYSQLControl {
        static final Log logger = LogFactory.getLog(MYSQLControl.class);
        static DataSource ds = MyDataSource.getDataSource("jdbc:mysql://127.0.0.1:3306/datacollection");
        static QueryRunner qr = new QueryRunner(ds);
        //第一类方法
        public static void executeUpdate(String sql){
            try {
                qr.update(sql);
            } catch (SQLException e) {
                logger.error(e);
            }
        }
        //按照SQL查询单个结果
        public static Object getScalaBySQL ( String sql ){
    
            ResultSetHandler<Object> h = new ScalarHandler<Object>(1);
            Object obj = null;
            try {
                obj = qr.query(sql, h);
            } catch (SQLException e) {
                e.printStackTrace();
            }
            return obj;
    
        }
        //按照SQL查询多个结果
        public static <T> List<T> getListInfoBySQL (String sql, Class<T> type ){
            List<T> list = null;
            try {
                list = qr.query(sql,new BeanListHandler<T>(type));
            } catch (SQLException e) {
                e.printStackTrace();
            }
            return list;
        }
        //查询一列
        public static List<Object> getListOneBySQL (String sql,String id){
            List<Object> list=null;
    
            try {
                list = (List<Object>) qr.query(sql, new ColumnListHandler(id));
            } catch (SQLException e) {
                e.printStackTrace();
            }
            return list;
        }
        //此种数据库操作方法需要优化
        public static int insertoilStocks ( List<ExtMarketOilStockModel> oilstocks ) {
    
            Object[][] params = new Object[oilstocks.size()][17];
            int c = 0;  //success number of update
            int[] sum;
            for ( int i = 0; i < oilstocks.size(); i++ ){
                params[i][0] = oilstocks.get(i).getDate();
                params[i][1] = oilstocks.get(i).getStock_id();
                params[i][2] = oilstocks.get(i).getStock_name();
                params[i][3] = oilstocks.get(i).getStock_price();
                params[i][4] = oilstocks.get(i).getStock_change();
                params[i][5] = oilstocks.get(i).getStock_range();
                params[i][6] = oilstocks.get(i).getStock_amplitude();
                params[i][7] = oilstocks.get(i).getStock_trading_number();
                params[i][8] = oilstocks.get(i).getStock_trading_value();
                params[i][9] = oilstocks.get(i).getStock_yesterdayfinish_price();
                params[i][10] = oilstocks.get(i).getStock_todaystart_price();
                params[i][11] = oilstocks.get(i).getStock_max_price();
                params[i][12] = oilstocks.get(i).getStock_min_price();
                params[i][13] = oilstocks.get(i).getStock_fiveminuate_change();
                params[i][14] = oilstocks.get(i).getCraw_time();
                params[i][15] = null;
                params[i][16] = null;
            }
    
            QueryRunner qr = new QueryRunner(ds);
            try {
                sum = qr.batch("INSERT INTO `datacollection`.`ext_market_oil_stock` VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", params);
            } catch (SQLException e) {
                System.out.println(e);
            }
            System.out.println("石油数据入库完毕");
    
            return c;
    
        }
        //此种数据库操作方法需要优化
        public static int insertcarStocks ( List<ExtMarketOilStockModel> carstocks ) {
    
            int c = 0;  //success number of update
            int[] sum;
            Object[][] params1 = new Object[carstocks.size()][17];
            int c1 = 0; //success number of update
            for ( int i = 0; i < carstocks.size(); i++ ){
                params1[i][0] = carstocks.get(i).getDate();
                params1[i][1] = carstocks.get(i).getStock_id();
                params1[i][2] = carstocks.get(i).getStock_name();
                params1[i][3] = carstocks.get(i).getStock_price();
                params1[i][4] = carstocks.get(i).getStock_change();
                params1[i][5] = carstocks.get(i).getStock_range();
                params1[i][6] = carstocks.get(i).getStock_amplitude();
                params1[i][7] = carstocks.get(i).getStock_trading_number();
                params1[i][8] = carstocks.get(i).getStock_trading_value();
                params1[i][9] = carstocks.get(i).getStock_yesterdayfinish_price();
                params1[i][10] = carstocks.get(i).getStock_todaystart_price();
                params1[i][11] = carstocks.get(i).getStock_max_price();
                params1[i][12] = carstocks.get(i).getStock_min_price();
                params1[i][13] = carstocks.get(i).getStock_fiveminuate_change();
                params1[i][14] = carstocks.get(i).getCraw_time();
                params1[i][15] = null;
                params1[i][16] = null;
            }
            QueryRunner qr = new QueryRunner(ds);
            try {
            //插入的数据表及数据
                sum = qr.batch("INSERT INTO `datacollection`.`ext_market_car_stock` VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", params1);
    
            } catch (SQLException e) {
                System.out.println(e);
            }
            System.out.println("汽车数据入库完毕");
    
            return c;
    
        }
    
    }

**这样按道理整个爬虫，程序就写完了，运行main方法就行了。如下图，为main方法获取数据的部分结果。**

![这里写图片描述][SouthEast 4]

# 问题所在 #

**问题1：针对股票这种数据，每周1到周五都会发布相关股票数据，那么如何每天定时定点让程序自动的去抓取，而不是手工每天运行一下呢？**

**问题二：股票节假日，是不会开盘的，当网页中存在此数据，即网页中的显示，没有时间标签。针对此，又该如何处理呢？**

首先，我带大家来看看我的数据库设计。

![这里写图片描述][SouthEast 5]

# 解决方法 #

这里使用Quartz实线定期运行程序，即上面提的第一个问题。([http://blog.csdn.net/qy20115549/article/details/52723907][http_blog.csdn.net_qy20115549_article_details_52723907])。  
针对第二个问题使用是：**即如何判断当天股票不开盘，采用的方法是从数据库中随机抽取三个股票(上次时间的，如今天是1月21日，周六，随机从数据库中抽取1月20日的三只股票。将1月20日的三只股票与今天相同id的股票价格进行比较，如果三个股票的价格都相同，则判断，改天为节假日，股票价格没有变动，无需将数据插入数据库)。**

## job ##

package job;
    
    import java.util.ArrayList;
    import java.util.List;
    import org.quartz.Job; 
    import org.quartz.JobExecutionContext; 
    import org.quartz.JobExecutionException;
    import db.MYSQLControl;
    import model.ExtMarketOilStockModel;
    import parse.ExtMarketOilStockParse;
    import timecontrol.TimeControl;
     /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public class ExtMarketOilStockJob implements Job {  
    
        @Override 
        public void execute(JobExecutionContext arg0) throws JobExecutionException {
            //获取上次的插入股票日期，加入判断是否为节假日
            List<ExtMarketOilStockModel> randomlist = MYSQLControl.getListInfoBySQL("select stock_id,stock_price,stock_change from ext_market_oil_stock where date = (select date from ext_market_oil_stock order by date desc limit 1) ",ExtMarketOilStockModel.class);
            //表格更新时间
    
            List<String> urloillist=new ArrayList<String>();
            List<String> urlcarlist=new ArrayList<String>();
            List<ExtMarketOilStockModel> oilstocks=new ArrayList<ExtMarketOilStockModel>();
            List<ExtMarketOilStockModel> carstocks=new ArrayList<ExtMarketOilStockModel>();
            String url1="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=1&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.13204790262127375";
            String url2="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=2&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.6972178580603532";
            urloillist.add(url1);
            urloillist.add(url2);
            int judge=0;
            for (int i = 0; i < urloillist.size(); i++) {
                try {
                    oilstocks=ExtMarketOilStockParse.parseurl(urloillist.get(i));
                } catch (Exception e) {
                    e.printStackTrace();
                }
    
                for (int j = 0; j < oilstocks.size(); j++) {
                    String stock_id=oilstocks.get(j).getStock_id();
                    float stock_price=oilstocks.get(j).getStock_price();
                    if (stock_id.equals(randomlist.get(0).getStock_id())) {
                        if (stock_price==randomlist.get(0).getStock_price()) {
                            judge++;
                        }
                    }
                }
                for (int j = 0; j < oilstocks.size(); j++) {
                    String stock_id=oilstocks.get(j).getStock_id();
                    float stock_price=oilstocks.get(j).getStock_price();
                    if (stock_id.equals(randomlist.get(1).getStock_id())) {
                        if (stock_price==randomlist.get(1).getStock_price()) {
                            judge++;
                        }
                    }
                }
                for (int j = 0; j < oilstocks.size(); j++) {
                    String stock_id=oilstocks.get(j).getStock_id();
                    float stock_price=oilstocks.get(j).getStock_price();
                    if (stock_id.equals(randomlist.get(2).getStock_id())) {
                        if (stock_price==randomlist.get(2).getStock_price()) {
                            judge++;
                        }
                    }
                }
                if (judge!=3) {
                    MYSQLControl.insertoilStocks(oilstocks);
                }
            }
            if (judge!=3) {
                for (int i = 1; i <6; i++) {
                    String urli="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04811&sty=FCOIATA&sortType=C&sortRule=-1&page="+i+"&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.23492960370783944";
                    urlcarlist.add(urli);
                }
                for (int i = 0; i < urlcarlist.size(); i++) {
                    try {
                        carstocks=ExtMarketOilStockParse.parseurl(urlcarlist.get(i));
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                    MYSQLControl.insertcarStocks(carstocks);
                }
            }
    
        } 
    
    }

## jobmain ##

如下，控制的时间是每周一到周五，8点39执行job，即每天都去抓取数据。

package jobmain;
    import static org.quartz.CronScheduleBuilder.cronSchedule;
    import static org.quartz.JobBuilder.newJob;
    import static org.quartz.TriggerBuilder.newTrigger;
    import java.text.SimpleDateFormat;
    import java.util.Date;
    import org.quartz.CronTrigger;
    import org.quartz.JobDetail;
    import org.quartz.Scheduler;
    import org.quartz.SchedulerFactory;
    import org.quartz.impl.StdSchedulerFactory;
    import job.ExtMarketOilStockJob;
     /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
    public class ExtMarketOilStockJobMain { 
    
        public void go() throws Exception { 
            // 首先，必需要取得一个Scheduler的引用 
            SchedulerFactory sf = new StdSchedulerFactory(); 
            Scheduler sched = sf.getScheduler(); 
            //jobs可以在scheduled的sched.start()方法前被调用 
            JobDetail job = newJob(ExtMarketOilStockJob.class).withIdentity("stockjob", "stockgroup").build(); 
            //每周一到周五8点39开始执行job
            CronTrigger trigger = newTrigger().withIdentity("stocktrigger", "stockgroup").withSchedule(cronSchedule("0 39 20 ? * MON-FRI")).build(); 
            Date ft = sched.scheduleJob(job, trigger); 
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss SSS"); 
            System.out.println(job.getKey() + " 已被安排执行于: " + sdf.format(ft) + "，并且以如下重复规则重复执行: " + trigger.getCronExpression()); 
            sched.start(); 
        } 
        public static void main(String[] args) throws Exception { 
            ExtMarketOilStockJobMain maingo = new ExtMarketOilStockJobMain(); 
            maingo.go(); 
        } 
    
    }

**运行jobmain中的类，便可以实现每天定点爬取数据。**

[http_quote.eastmoney.com_center_list.html_28002481_0_2]: http://quote.eastmoney.com/center/list.html#28002481_0_2
[http_quote.eastmoney.com_center_list.html_28002464_0_2]: http://quote.eastmoney.com/center/list.html#28002464_0_2
[SouthEast]: /images/20220711/3065e4cef8b644a58f1b10cb28fa97ae.png
[http_blog.csdn.net_column_details_14269.html]: http://blog.csdn.net/column/details/14269.html
[SouthEast 1]: /images/20220711/c3d297acd09a4048b930c0b08828c591.png
[SouthEast 2]: /images/20220711/90f057839b3042e8aa45ed85395c26d0.png
[SouthEast 3]: /images/20220711/363ba45da5c64978a81868af22b4ff8d.png
[http_blog.csdn.net_qy20115549_article_details_52203722]: http://blog.csdn.net/qy20115549/article/details/52203722
[SouthEast 4]: /images/20220711/e247dd1a85d94616b57c189f266c8e03.png
[SouthEast 5]: /images/20220711/7d44adfa3c14476f9d06ba51702aa3a6.png
[http_blog.csdn.net_qy20115549_article_details_52723907]: http://blog.csdn.net/qy20115549/article/details/52723907