python爬虫-初次接触

待我称王封你为后i 2022-07-14 04:38 198阅读 0赞

说来惭愧，学习程序这么多年，居然都没有为自己怎么编写过程序，之前一直鄙视python一类的脚本语言，想来是多么的无知呀。。

最近觉得必须为自己写写程序了，不能一辈子只为别人服务是吧。

我理解的网络爬虫，程序自动化的抓取人们关心的网页内容，还可以根据超级链接进一步的往下抓数据，这个就是爬虫。

我的第一个网络爬虫就是获取股票数据

[https://gupiao.baidu.com/][https_gupiao.baidu.com]

获取百度股市通里面的股票的历史成交数据

打开网址 [ https://gupiao.baidu.com/stock/sz300369.html][https_gupiao.baidu.com_stock_sz300369.html] 可以查看到我司股票。

使用google浏览器，F12 选择Network

![Center][]

然后选择日线 点击拉长K线，可以在Network中看到有数据流。选择最大的数据的一个流，可以就看到请求的数据了。

![Image 1][]

![Center 1][]

![Image 1][]

![Center 2][]

这样就可以获取到产生K线图的，数据来源了，使用的GET然后对应的URL是

[https://gupiao.baidu.com/api/stocks/stockdaybar?from=pc&os\_ver=1&cuid=xxx&vv=100&format=json&stock\_code=sz300369&step=3&start=20160308&count=160&fq\_type=no&timestamp=1480252213081][https_gupiao.baidu.com_api_stocks_stockdaybar_from_pc_os_ver_1_cuid_xxx_vv_100_format_json_stock_code_sz300369_step_3_start_20160308_count_160_fq_type_no_timestamp_1480252213081]

到此~~就获取到数据的URL，这个获取方式比较简单就是普通的URL方式。我们只需要通过使用python来请求这个数据就好了。

然后参考这篇文章

[http://python.jobbole.com/81336/][http_python.jobbole.com_81336]

![Center 3][]

![Image 1][]

依葫芦画瓢，就得到如下的代码

import urllib
        import urllib2
        values={}
        values['from'] ='pc'
        values['os_ver'] ='1'
        values['cuid'] ='xxx'
        values['vv'] ='100'
        values['format'] ='json'
        values['stock_code'] ='sz300369'
        values['step'] ='3'
        values['start'] =start
        values['count'] ='320'
        values['fq_type'] ='no'
        values['timestamp'] ='1480063069006'
    
    
        data = urllib.urlencode(values)
        url = "https://gupiao.baidu.com/api/stocks/stockdaybar"
        geturl = url + "?"+data
        print(geturl)
        request = urllib2.Request(geturl)
    	response = urllib2.urlopen(request)

完成获取所有股票数据的代码如下

从文件123.txt读出所有股票的代码。仅限上证

__author__ = 'bestboyxie'
    
    
    import urllib
    import urllib2
    import cookielib
    import codecs
    import json
    import re
    import thread
    import time
    #https://gupiao.baidu.com/api/stocks/stockdaybar?from=pc&os_ver=1&cuid=xxx&vv=100&format=json&stock_code=sh000001&step=3&start=20150710&count=320&fq_type=no×tamp=1480054552865
    #https://gupiao.baidu.com/api/stocks/stockdaybar?from=pc&os_ver=1&cuid=xxx&vv=100&format=json&stock_code=sz300369&step=3&start=20150706&count=320&fq_type=no×tamp=1480055288950
    def getdata(start,stock_code):
        values={}
        values['from'] ='pc'
        values['os_ver'] ='1'
        values['cuid'] ='xxx'
        values['vv'] ='100'
        values['format'] ='json'
        values['stock_code'] =stock_code
        values['step'] ='3'
        values['start'] =start
        values['count'] ='320'
        values['fq_type'] ='no'
        values['timestamp'] ='1480063069006'
    
    
        data = urllib.urlencode(values)
        url = "https://gupiao.baidu.com/api/stocks/stockdaybar"
        geturl = url + "?"+data
        print(geturl)
        request = urllib2.Request(geturl)
    
    
        cookie = cookielib.CookieJar()
        handler=urllib2.HTTPCookieProcessor(cookie)
        opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie))
    
        response = opener.open(request)
    
        #print response.read()
        output= codecs.open(stock_code+"_stockdaybar.txt", "a+", "utf-8")
        response_str=response.read()
        mindate = int(values['start'])
        jsona = json.loads(response_str,'utf-8')
        if 'mashData' in jsona:
            mashDatas = jsona['mashData']
            output.write(json.dumps(jsona['mashData']))
            for mashData in mashDatas:
                date = int(mashData['date'])
                if mindate - date >0:
                    mindate = date
            print(mindate)
        output.close()
        return mindate
    def filetoarray(filename):
        IDXfile = open(filename, 'r')
        fileread=IDXfile.read()
        IDXfile.close()
        p = re.compile(r"\d+")
        l = p.findall(fileread)
        return l
    def down2file(threadName, stock):
        date = 20161125
        name =""
        if stock > 600000 and  stock < 604000 :
            name ='sh'+str(stock)
        else :
            return
        last_data = 0
        while last_data!= date:
            last_data = date
            date = getdata(last_data,name)
    stock_all = filetoarray("123.txt")
    for stock in stock_all:
        thread.start_new_thread(down2file,(str(stock),int(stock)))
        time.sleep(1)
    #print response.read()

其中123.txt的格式如下

浦发银行(600000)邯郸钢铁(600001)ST东北高(600003)白云机场(600004)武钢股份(600005)东风汽车(600006)中国国贸(600007)首创股份(600008)上海机场(600009)包钢股份(600010)华能国际(600011)皖通高速(600012)华夏银行(600015)民生银行(600016)日照港(600017)上港集团(600018)宝钢股份(600019)中原高速(600020)上海电力(600021)山东钢铁(600022)浙能电力(600023)中海发展(600026)华电国际(600027)中国石化(600028)南方航空(600029)中信证券(600030)三一重工(600031)福建高速(600033)楚天高速(600035)招商银行(600036)歌华有线(600037)中直股份(600038)四川路桥(600039)保利地产(600048)中国联通(600050)宁波联合(600051)浙江广厦(600052)九鼎投资(600053)黄山旅游(600054)万东医疗(600055)中国医药(600056)象屿股份(600057)五矿发展(600058)古越龙山(600059)海信电器(600060)国投安信(600061)华润双鹤(600062)皖维高新(600063)南京高科(600064)宇通客车(600066)冠城大通(600067) 葛洲坝(600068) 银鸽投资(600069)

参考

http://python.jobbole.com/81359/

[https_gupiao.baidu.com]: https://gupiao.baidu.com/
[https_gupiao.baidu.com_stock_sz300369.html]: https://gupiao.baidu.com/stock/sz300369.html
[Center]: /images/20220714/8cd68b1d0c5e499b9f2cb27b9d5879c7.png
[Image 1]: 
[Center 1]: /images/20220714/fcbaf6577d9943a5a7e1e2751f49905e.png
[Center 2]: /images/20220714/f734945563ee48a0bb715aaa91137b9b.png
[https_gupiao.baidu.com_api_stocks_stockdaybar_from_pc_os_ver_1_cuid_xxx_vv_100_format_json_stock_code_sz300369_step_3_start_20160308_count_160_fq_type_no_timestamp_1480252213081]: https://gupiao.baidu.com/api/stocks/stockdaybar?from=pc&os_ver=1&cuid=xxx&vv=100&format=json&stock_code=sz300369&step=3&start=20160308&count=160&fq_type=no&timestamp=1480252213081
[http_python.jobbole.com_81336]: http://python.jobbole.com/81336/
[Center 3]: /images/20220714/af9cb5b8426548f4bec08bfd45855119.png