python 数据预处理1

短命女 2023-05-30 04:59 31阅读 0赞

通过[Tushare][]或者[Tushare pro][]获取股票清单，但是有3707家企业，怎么挑选股票，我自己瞎琢磨。  
**1 根据各板块选择龙头股票**  
[mysql分组排序取最大、最小、最新、前N条记录][mysql_N]，下图是根据行业、总市值来判断。

select st1.code,st1.name,sb1.industry,st1.mktcap,sb1.market,st1.trade,st1.per,st1.pb
    from stock_today st1
    inner join stock_basic sb1 on sb1.symbol=st1.code
    where 3> (
    select count(*)
    from stock_today st2
    inner join stock_basic sb2 on sb2.symbol=st2.code
    where sb1.industry=sb2.industry and sb1.market=sb2.market and st2.mktcap>st1.mktcap
    )
    order by sb1.industry,st1.mktcap

**2 市盈率Matplotlib作图**  
参考[Matplotlib - 箱线图、箱型图 boxplot () 所有用法详解][Matplotlib - _ boxplot _],[python中matplotlib的颜色及线条控制的示例][python_matplotlib]，[使用sklearn预处理数据之标准化、归一化、正则化][sklearn]

import numpy as np
    from matplotlib import pyplot as plt
    from sklearn.preprocessing import StandardScaler
    import pandas as pd
    
    def get_outlier(x,y,upper,lower):
        o_x = []
        o_y = []
        for i in range(0,len(x)):
            if y[i]< lower or y[i]>upper:
                o_x.append(x[i])
                o_y.append(y[i])
            else:
                continue
        return o_x,o_y
    # figsize:指定figure的宽和高，单位为英寸, 1英寸等于2.5cm,A4纸是 21*30cm的纸张 
    # dpi参数指定绘图对象的分辨率，dpi默认为80
    fig = plt.figure(figsize=(15,15),dpi=60)
    # 数据
    per=[35.676,33.163,53.553,38.374,36.239,18.501,43.831,16.49,32.333,42.884,21.019,32.684,50.282,53.996,23.175,34.972,21.294,24.31]
    per=np.array(per)
    mean = round(per.mean(),2)
    x = np.arange(1,19,1)
    # 散点图
    # add_subplot，参数顺序：行数、列数，位置，221，也可以协程2,2,1
    ax = fig.add_subplot(221)
    ax.scatter(x, per, marker = 'o', color = 'green', s = 40, label = 'per')
    ax.axhline(y=mean,label=str(mean))
    ax.legend(loc='best')
    # z-score标准化
    ss = StandardScaler()
    # fit:用于计算训练数据的均值和方差， 后面就会用均值和方差来转换训练数据
    # transform:很显然，它只是进行转换，只是把训练数据转换成标准的正态分布
    # fit_transform:不仅计算训练数据的均值和方差，还会基于计算出来的均值和方差来转换训练数据，从而把数据转换成标准的正太分布
    # reshape中-1代表行数未知，列数为1
    ss_per = ss.fit_transform(per.reshape(-1,1))
    ax = fig.add_subplot(222)
    # sklearn中二维数组结构，numpy https://www.zhangshengrong.com/p/bYXxZKwlaZ/，减少内存消耗，需要做一下转换
    ax.scatter(x,ss_per.reshape(1,-1)[0], marker = 'o', color = 'green', s = 40, label = 'Third')
    # 箱型图
    ax = fig.add_subplot(223)
    ax.boxplot(per, showmeans=True)
    df_per = pd.DataFrame(per)
    # pandas.DataFrame.describe，返回数据的统计特征
    # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html
    statistics = df_per.describe()
    # https://www.cnblogs.com/linux-wangkun/p/5903380.html
    IQR = statistics.loc['75%']-statistics.loc['25%']
    QL = statistics.loc['25%']
    QU = statistics.loc['75%']
    threshold_upper = QL - 1.5 * IQR
    threshold_lower = QU + 1.5 * IQR
    # https://www.cnblogs.com/linux-wangkun/p/5903380.html
    # 注意pandas 数据结构之Series
    o_x,o_y = get_outlier(x,per,QU[0],QL[0])
    ax = fig.add_subplot(224)
    ax.plot(x,per,marker='+')
    ax.axhline(y=threshold_upper[0])
    ax.axhline(y=QL[0],color='r')
    ax.axhline(y=QU[0],color='c')
    ax.axhline(y=threshold_lower[0])
    # ax.plot(o_x,o_y)
    for i in range(len(o_x)):
        ax.annotate(o_y[i],xy=(o_x[i],o_y[i]),xytext=(o_x[i],o_y[i]))
    # 显示
    plt.show()

从下图可以看到matplotlib虽然可以画箱型图，但是图形意义只是直观，而作用并不大，至少还需要pandas分析。  
数据标准化后，数据的分布并不会发生变化。  
![1][]  
待续。。

[Tushare]: http://tushare.org/trading.html#id4
[Tushare pro]: https://tushare.pro/document/2?doc_id=25
[mysql_N]: https://blog.csdn.net/yige9394/article/details/79481706
[Matplotlib - _ boxplot _]: https://blog.csdn.net/weixin_40683253/article/details/87857194
[python_matplotlib]: https://www.jb51.net/article/136566.htm
[sklearn]: https://blog.csdn.net/qq_24831889/article/details/83344638
[1]: https://img-blog.csdnimg.cn/20191108112504883.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70