二手车预测：task2：数据分布

刺骨的言语ヽ痛彻心扉 2023-07-17 15:41 50阅读 0赞

## 2.1 EDA目标 ##

*  EDA的价值主要在于熟悉数据集，了解数据集，对数据集进行验证来确定所获得数据集可以用于接下来的机器学习或者深度学习使用。
 *  当了解了数据集之后我们下一步就是要去了解变量间的相互关系以及变量与预测值之间的存在关系。
 *  引导数据科学从业者进行数据处理以及特征工程的步骤,使数据集的结构和特征集让接下来的预测问题更加可靠。
 *  完成对于数据的探索性分析，并对于数据进行一些图表或者文字总结并打卡。

## 2.2 内容介绍 ##

1.  载入各种数据科学以及可视化库:
    
     *  数据科学库 pandas、numpy、scipy；
     *  可视化库 matplotlib、seabon；
     *  其他；
2.  载入数据：
    
     *  载入训练集和测试集；
     *  简略观察数据(head()+shape)；
3.  数据总览:
    
     *  通过describe()来熟悉数据的相关统计量
     *  通过info()来熟悉数据类型
4.  判断数据缺失和异常
    
     *  查看每列的存在nan情况
     *  异常值检测
5.  了解预测值的分布
    
     *  总体分布概况（无界约翰逊分布等）
     *  查看skewness and kurtosis
     *  查看预测值的具体频数
6.  特征分为类别特征和数字特征，并对类别特征查看unique分布
7.  数字特征分析
    
     *  相关性分析
     *  查看几个特征得 偏度和峰值
     *  每个数字特征得分布可视化
     *  数字特征相互之间的关系可视化
     *  多变量互相回归关系可视化
8.  类型特征分析
    
     *  unique分布
     *  类别特征箱形图可视化
     *  类别特征的小提琴图可视化
     *  类别特征的柱形图可视化类别
     *  特征的每个类别频数可视化(count\_plot)
9.  用pandas\_profiling生成数据报告

# 导入包 #

#coding:utf-8
    #导入warnings包，利用过滤器来实现忽略警告语句。
    !pip install missingno
    import warnings
    warnings.filterwarnings('ignore')
    
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns
    import missingno as msno# 缺失值分析的代码

Collecting missingno
      Downloading https://files.pythonhosted.org/packages/2b/de/6e4dd6d720c49939544352155dc06a08c9f7e4271aa631a559dfbeaaf9d4/missingno-0.4.2-py3-none-any.whl
    Requirement already satisfied: numpy in /home/ach/anaconda3/lib/python3.7/site-packages (from missingno) (1.17.2)
    Requirement already satisfied: matplotlib in /home/ach/anaconda3/lib/python3.7/site-packages (from missingno) (3.1.1)
    Requirement already satisfied: scipy in /home/ach/anaconda3/lib/python3.7/site-packages (from missingno) (1.3.1)
    Requirement already satisfied: seaborn in /home/ach/anaconda3/lib/python3.7/site-packages (from missingno) (0.9.0)
    Requirement already satisfied: cycler>=0.10 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib->missingno) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib->missingno) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib->missingno) (2.4.2)
    Requirement already satisfied: python-dateutil>=2.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib->missingno) (2.8.0)
    Requirement already satisfied: pandas>=0.15.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from seaborn->missingno) (0.25.1)
    Requirement already satisfied: six in /home/ach/anaconda3/lib/python3.7/site-packages (from cycler>=0.10->matplotlib->missingno) (1.12.0)
    Requirement already satisfied: setuptools in /home/ach/anaconda3/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->missingno) (41.4.0)
    Requirement already satisfied: pytz>=2017.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas>=0.15.2->seaborn->missingno) (2019.3)
    Installing collected packages: missingno
    Successfully installed missingno-0.4.2

# 导入数据 #

## 1) 载入训练集和测试集；
    Train_data = pd.read_csv('/home/ach/下载/数据挖掘/used_car_train_20200313.csv', sep=' ')
    Test_data = pd.read_csv('/home/ach/下载/数据挖掘/used_car_testA_20200313.csv', sep=' ')

### 所有特征集均脱敏处理(方便大家观看) ###

*  name - 汽车编码
 *  regDate - 汽车注册时间
 *  model - 车型编码
 *  brand - 品牌
 *  bodyType - 车身类型
 *  fuelType - 燃油类型
 *  gearbox - 变速箱
 *  power - 汽车功率
 *  kilometer - 汽车行驶公里
 *  notRepairedDamage - 汽车有尚未修复的损坏
 *  regionCode - 看车地区编码
 *  seller - 销售方
 *  offerType - 报价类型
 *  creatDate - 广告发布时间
 *  price - 汽车价格
 *  v\_0’, ‘v\_1’, ‘v\_2’, ‘v\_3’, ‘v\_4’, ‘v\_5’, ‘v\_6’, ‘v\_7’, ‘v\_8’, ‘v\_9’, ‘v\_10’, ‘v\_11’, ‘v\_12’, ‘v\_13’,‘v\_14’（根据汽车的评论、标签等大量信息得到的embedding向量）【人工构造 匿名特征】

#The tail() function is used to return the last n rows. 
    ## 2) 简略观察数据(head()+shape)
    Train_data.head().append(Train_data.tail())

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>SaleID</th> 
   <th>name</th> 
   <th>regDate</th> 
   <th>model</th> 
   <th>brand</th> 
   <th>bodyType</th> 
   <th>fuelType</th> 
   <th>gearbox</th> 
   <th>power</th> 
   <th>kilometer</th> 
   <th>...</th> 
   <th>v_5</th> 
   <th>v_6</th> 
   <th>v_7</th> 
   <th>v_8</th> 
   <th>v_9</th> 
   <th>v_10</th> 
   <th>v_11</th> 
   <th>v_12</th> 
   <th>v_13</th> 
   <th>v_14</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>0</td> 
   <td>0</td> 
   <td>736</td> 
   <td>20040402</td> 
   <td>30.0</td> 
   <td>6</td> 
   <td>1.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>60</td> 
   <td>12.5</td> 
   <td>...</td> 
   <td>0.235676</td> 
   <td>0.101988</td> 
   <td>0.129549</td> 
   <td>0.022816</td> 
   <td>0.097462</td> 
   <td>-2.881803</td> 
   <td>2.804097</td> 
   <td>-2.420821</td> 
   <td>0.795292</td> 
   <td>0.914762</td> 
  </tr> 
  <tr> 
   <td>1</td> 
   <td>1</td> 
   <td>2262</td> 
   <td>20030301</td> 
   <td>40.0</td> 
   <td>1</td> 
   <td>2.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>0</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.264777</td> 
   <td>0.121004</td> 
   <td>0.135731</td> 
   <td>0.026597</td> 
   <td>0.020582</td> 
   <td>-4.900482</td> 
   <td>2.096338</td> 
   <td>-1.030483</td> 
   <td>-1.722674</td> 
   <td>0.245522</td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>2</td> 
   <td>14874</td> 
   <td>20040403</td> 
   <td>115.0</td> 
   <td>15</td> 
   <td>1.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>163</td> 
   <td>12.5</td> 
   <td>...</td> 
   <td>0.251410</td> 
   <td>0.114912</td> 
   <td>0.165147</td> 
   <td>0.062173</td> 
   <td>0.027075</td> 
   <td>-4.846749</td> 
   <td>1.803559</td> 
   <td>1.565330</td> 
   <td>-0.832687</td> 
   <td>-0.229963</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>3</td> 
   <td>71865</td> 
   <td>19960908</td> 
   <td>109.0</td> 
   <td>10</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>193</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.274293</td> 
   <td>0.110300</td> 
   <td>0.121964</td> 
   <td>0.033395</td> 
   <td>0.000000</td> 
   <td>-4.509599</td> 
   <td>1.285940</td> 
   <td>-0.501868</td> 
   <td>-2.438353</td> 
   <td>-0.478699</td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>4</td> 
   <td>111080</td> 
   <td>20120103</td> 
   <td>110.0</td> 
   <td>5</td> 
   <td>1.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>68</td> 
   <td>5.0</td> 
   <td>...</td> 
   <td>0.228036</td> 
   <td>0.073205</td> 
   <td>0.091880</td> 
   <td>0.078819</td> 
   <td>0.121534</td> 
   <td>-1.896240</td> 
   <td>0.910783</td> 
   <td>0.931110</td> 
   <td>2.834518</td> 
   <td>1.923482</td> 
  </tr> 
  <tr> 
   <td>149995</td> 
   <td>149995</td> 
   <td>163978</td> 
   <td>20000607</td> 
   <td>121.0</td> 
   <td>10</td> 
   <td>4.0</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>163</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.280264</td> 
   <td>0.000310</td> 
   <td>0.048441</td> 
   <td>0.071158</td> 
   <td>0.019174</td> 
   <td>1.988114</td> 
   <td>-2.983973</td> 
   <td>0.589167</td> 
   <td>-1.304370</td> 
   <td>-0.302592</td> 
  </tr> 
  <tr> 
   <td>149996</td> 
   <td>149996</td> 
   <td>184535</td> 
   <td>20091102</td> 
   <td>116.0</td> 
   <td>11</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>125</td> 
   <td>10.0</td> 
   <td>...</td> 
   <td>0.253217</td> 
   <td>0.000777</td> 
   <td>0.084079</td> 
   <td>0.099681</td> 
   <td>0.079371</td> 
   <td>1.839166</td> 
   <td>-2.774615</td> 
   <td>2.553994</td> 
   <td>0.924196</td> 
   <td>-0.272160</td> 
  </tr> 
  <tr> 
   <td>149997</td> 
   <td>149997</td> 
   <td>147587</td> 
   <td>20101003</td> 
   <td>60.0</td> 
   <td>11</td> 
   <td>1.0</td> 
   <td>1.0</td> 
   <td>0.0</td> 
   <td>90</td> 
   <td>6.0</td> 
   <td>...</td> 
   <td>0.233353</td> 
   <td>0.000705</td> 
   <td>0.118872</td> 
   <td>0.100118</td> 
   <td>0.097914</td> 
   <td>2.439812</td> 
   <td>-1.630677</td> 
   <td>2.290197</td> 
   <td>1.891922</td> 
   <td>0.414931</td> 
  </tr> 
  <tr> 
   <td>149998</td> 
   <td>149998</td> 
   <td>45907</td> 
   <td>20060312</td> 
   <td>34.0</td> 
   <td>10</td> 
   <td>3.0</td> 
   <td>1.0</td> 
   <td>0.0</td> 
   <td>156</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.256369</td> 
   <td>0.000252</td> 
   <td>0.081479</td> 
   <td>0.083558</td> 
   <td>0.081498</td> 
   <td>2.075380</td> 
   <td>-2.633719</td> 
   <td>1.414937</td> 
   <td>0.431981</td> 
   <td>-1.659014</td> 
  </tr> 
  <tr> 
   <td>149999</td> 
   <td>149999</td> 
   <td>177672</td> 
   <td>19990204</td> 
   <td>19.0</td> 
   <td>28</td> 
   <td>6.0</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>193</td> 
   <td>12.5</td> 
   <td>...</td> 
   <td>0.284475</td> 
   <td>0.000000</td> 
   <td>0.040072</td> 
   <td>0.062543</td> 
   <td>0.025819</td> 
   <td>1.978453</td> 
   <td>-3.179913</td> 
   <td>0.031724</td> 
   <td>-1.483350</td> 
   <td>-0.342674</td> 
  </tr> 
 </tbody> 
</table>

10 rows × 31 columns

Train_data.shape

(150000, 31)

Test_data.head().append(Test_data.tail())

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>SaleID</th> 
   <th>name</th> 
   <th>regDate</th> 
   <th>model</th> 
   <th>brand</th> 
   <th>bodyType</th> 
   <th>fuelType</th> 
   <th>gearbox</th> 
   <th>power</th> 
   <th>kilometer</th> 
   <th>...</th> 
   <th>v_5</th> 
   <th>v_6</th> 
   <th>v_7</th> 
   <th>v_8</th> 
   <th>v_9</th> 
   <th>v_10</th> 
   <th>v_11</th> 
   <th>v_12</th> 
   <th>v_13</th> 
   <th>v_14</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>0</td> 
   <td>150000</td> 
   <td>66932</td> 
   <td>20111212</td> 
   <td>222.0</td> 
   <td>4</td> 
   <td>5.0</td> 
   <td>1.0</td> 
   <td>1.0</td> 
   <td>313</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.264405</td> 
   <td>0.121800</td> 
   <td>0.070899</td> 
   <td>0.106558</td> 
   <td>0.078867</td> 
   <td>-7.050969</td> 
   <td>-0.854626</td> 
   <td>4.800151</td> 
   <td>0.620011</td> 
   <td>-3.664654</td> 
  </tr> 
  <tr> 
   <td>1</td> 
   <td>150001</td> 
   <td>174960</td> 
   <td>19990211</td> 
   <td>19.0</td> 
   <td>21</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>75</td> 
   <td>12.5</td> 
   <td>...</td> 
   <td>0.261745</td> 
   <td>0.000000</td> 
   <td>0.096733</td> 
   <td>0.013705</td> 
   <td>0.052383</td> 
   <td>3.679418</td> 
   <td>-0.729039</td> 
   <td>-3.796107</td> 
   <td>-1.541230</td> 
   <td>-0.757055</td> 
  </tr> 
  <tr> 
   <td>2</td> 
   <td>150002</td> 
   <td>5356</td> 
   <td>20090304</td> 
   <td>82.0</td> 
   <td>21</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>109</td> 
   <td>7.0</td> 
   <td>...</td> 
   <td>0.260216</td> 
   <td>0.112081</td> 
   <td>0.078082</td> 
   <td>0.062078</td> 
   <td>0.050540</td> 
   <td>-4.926690</td> 
   <td>1.001106</td> 
   <td>0.826562</td> 
   <td>0.138226</td> 
   <td>0.754033</td> 
  </tr> 
  <tr> 
   <td>3</td> 
   <td>150003</td> 
   <td>50688</td> 
   <td>20100405</td> 
   <td>0.0</td> 
   <td>0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>160</td> 
   <td>7.0</td> 
   <td>...</td> 
   <td>0.260466</td> 
   <td>0.106727</td> 
   <td>0.081146</td> 
   <td>0.075971</td> 
   <td>0.048268</td> 
   <td>-4.864637</td> 
   <td>0.505493</td> 
   <td>1.870379</td> 
   <td>0.366038</td> 
   <td>1.312775</td> 
  </tr> 
  <tr> 
   <td>4</td> 
   <td>150004</td> 
   <td>161428</td> 
   <td>19970703</td> 
   <td>26.0</td> 
   <td>14</td> 
   <td>2.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>75</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.250999</td> 
   <td>0.000000</td> 
   <td>0.077806</td> 
   <td>0.028600</td> 
   <td>0.081709</td> 
   <td>3.616475</td> 
   <td>-0.673236</td> 
   <td>-3.197685</td> 
   <td>-0.025678</td> 
   <td>-0.101290</td> 
  </tr> 
  <tr> 
   <td>49995</td> 
   <td>199995</td> 
   <td>20903</td> 
   <td>19960503</td> 
   <td>4.0</td> 
   <td>4</td> 
   <td>4.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>116</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.284664</td> 
   <td>0.130044</td> 
   <td>0.049833</td> 
   <td>0.028807</td> 
   <td>0.004616</td> 
   <td>-5.978511</td> 
   <td>1.303174</td> 
   <td>-1.207191</td> 
   <td>-1.981240</td> 
   <td>-0.357695</td> 
  </tr> 
  <tr> 
   <td>49996</td> 
   <td>199996</td> 
   <td>708</td> 
   <td>19991011</td> 
   <td>0.0</td> 
   <td>0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>75</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.268101</td> 
   <td>0.108095</td> 
   <td>0.066039</td> 
   <td>0.025468</td> 
   <td>0.025971</td> 
   <td>-3.913825</td> 
   <td>1.759524</td> 
   <td>-2.075658</td> 
   <td>-1.154847</td> 
   <td>0.169073</td> 
  </tr> 
  <tr> 
   <td>49997</td> 
   <td>199997</td> 
   <td>6693</td> 
   <td>20040412</td> 
   <td>49.0</td> 
   <td>1</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>1.0</td> 
   <td>224</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.269432</td> 
   <td>0.105724</td> 
   <td>0.117652</td> 
   <td>0.057479</td> 
   <td>0.015669</td> 
   <td>-4.639065</td> 
   <td>0.654713</td> 
   <td>1.137756</td> 
   <td>-1.390531</td> 
   <td>0.254420</td> 
  </tr> 
  <tr> 
   <td>49998</td> 
   <td>199998</td> 
   <td>96900</td> 
   <td>20020008</td> 
   <td>27.0</td> 
   <td>1</td> 
   <td>0.0</td> 
   <td>0.0</td> 
   <td>1.0</td> 
   <td>334</td> 
   <td>15.0</td> 
   <td>...</td> 
   <td>0.261152</td> 
   <td>0.000490</td> 
   <td>0.137366</td> 
   <td>0.086216</td> 
   <td>0.051383</td> 
   <td>1.833504</td> 
   <td>-2.828687</td> 
   <td>2.465630</td> 
   <td>-0.911682</td> 
   <td>-2.057353</td> 
  </tr> 
  <tr> 
   <td>49999</td> 
   <td>199999</td> 
   <td>193384</td> 
   <td>20041109</td> 
   <td>166.0</td> 
   <td>6</td> 
   <td>1.0</td> 
   <td>NaN</td> 
   <td>1.0</td> 
   <td>68</td> 
   <td>9.0</td> 
   <td>...</td> 
   <td>0.228730</td> 
   <td>0.000300</td> 
   <td>0.103534</td> 
   <td>0.080625</td> 
   <td>0.124264</td> 
   <td>2.914571</td> 
   <td>-1.135270</td> 
   <td>0.547628</td> 
   <td>2.094057</td> 
   <td>-1.552150</td> 
  </tr> 
 </tbody> 
</table>

10 rows × 30 columns

Test_data.shape

(50000, 30)

# 数据概况总览 #

## 1) 通过describe()来熟悉数据的相关统计量
    Train_data.describe()

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>SaleID</th> 
   <th>name</th> 
   <th>regDate</th> 
   <th>model</th> 
   <th>brand</th> 
   <th>bodyType</th> 
   <th>fuelType</th> 
   <th>gearbox</th> 
   <th>power</th> 
   <th>kilometer</th> 
   <th>...</th> 
   <th>v_5</th> 
   <th>v_6</th> 
   <th>v_7</th> 
   <th>v_8</th> 
   <th>v_9</th> 
   <th>v_10</th> 
   <th>v_11</th> 
   <th>v_12</th> 
   <th>v_13</th> 
   <th>v_14</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>count</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>1.500000e+05</td> 
   <td>149999.000000</td> 
   <td>150000.000000</td> 
   <td>145494.000000</td> 
   <td>141320.000000</td> 
   <td>144019.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>...</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
   <td>150000.000000</td> 
  </tr> 
  <tr> 
   <td>mean</td> 
   <td>74999.500000</td> 
   <td>68349.172873</td> 
   <td>2.003417e+07</td> 
   <td>47.129021</td> 
   <td>8.052733</td> 
   <td>1.792369</td> 
   <td>0.375842</td> 
   <td>0.224943</td> 
   <td>119.316547</td> 
   <td>12.597160</td> 
   <td>...</td> 
   <td>0.248204</td> 
   <td>0.044923</td> 
   <td>0.124692</td> 
   <td>0.058144</td> 
   <td>0.061996</td> 
   <td>-0.001000</td> 
   <td>0.009035</td> 
   <td>0.004813</td> 
   <td>0.000313</td> 
   <td>-0.000688</td> 
  </tr> 
  <tr> 
   <td>std</td> 
   <td>43301.414527</td> 
   <td>61103.875095</td> 
   <td>5.364988e+04</td> 
   <td>49.536040</td> 
   <td>7.864956</td> 
   <td>1.760640</td> 
   <td>0.548677</td> 
   <td>0.417546</td> 
   <td>177.168419</td> 
   <td>3.919576</td> 
   <td>...</td> 
   <td>0.045804</td> 
   <td>0.051743</td> 
   <td>0.201410</td> 
   <td>0.029186</td> 
   <td>0.035692</td> 
   <td>3.772386</td> 
   <td>3.286071</td> 
   <td>2.517478</td> 
   <td>1.288988</td> 
   <td>1.038685</td> 
  </tr> 
  <tr> 
   <td>min</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>1.991000e+07</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.500000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>-9.168192</td> 
   <td>-5.558207</td> 
   <td>-9.639552</td> 
   <td>-4.153899</td> 
   <td>-6.546556</td> 
  </tr> 
  <tr> 
   <td>25%</td> 
   <td>37499.750000</td> 
   <td>11156.000000</td> 
   <td>1.999091e+07</td> 
   <td>10.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>75.000000</td> 
   <td>12.500000</td> 
   <td>...</td> 
   <td>0.243615</td> 
   <td>0.000038</td> 
   <td>0.062474</td> 
   <td>0.035334</td> 
   <td>0.033930</td> 
   <td>-3.722303</td> 
   <td>-1.951543</td> 
   <td>-1.871846</td> 
   <td>-1.057789</td> 
   <td>-0.437034</td> 
  </tr> 
  <tr> 
   <td>50%</td> 
   <td>74999.500000</td> 
   <td>51638.000000</td> 
   <td>2.003091e+07</td> 
   <td>30.000000</td> 
   <td>6.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>110.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.257798</td> 
   <td>0.000812</td> 
   <td>0.095866</td> 
   <td>0.057014</td> 
   <td>0.058484</td> 
   <td>1.624076</td> 
   <td>-0.358053</td> 
   <td>-0.130753</td> 
   <td>-0.036245</td> 
   <td>0.141246</td> 
  </tr> 
  <tr> 
   <td>75%</td> 
   <td>112499.250000</td> 
   <td>118841.250000</td> 
   <td>2.007111e+07</td> 
   <td>66.000000</td> 
   <td>13.000000</td> 
   <td>3.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>150.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.265297</td> 
   <td>0.102009</td> 
   <td>0.125243</td> 
   <td>0.079382</td> 
   <td>0.087491</td> 
   <td>2.844357</td> 
   <td>1.255022</td> 
   <td>1.776933</td> 
   <td>0.942813</td> 
   <td>0.680378</td> 
  </tr> 
  <tr> 
   <td>max</td> 
   <td>149999.000000</td> 
   <td>196812.000000</td> 
   <td>2.015121e+07</td> 
   <td>247.000000</td> 
   <td>39.000000</td> 
   <td>7.000000</td> 
   <td>6.000000</td> 
   <td>1.000000</td> 
   <td>19312.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.291838</td> 
   <td>0.151420</td> 
   <td>1.404936</td> 
   <td>0.160791</td> 
   <td>0.222787</td> 
   <td>12.357011</td> 
   <td>18.819042</td> 
   <td>13.847792</td> 
   <td>11.147669</td> 
   <td>8.658418</td> 
  </tr> 
 </tbody> 
</table>

8 rows × 30 columns

Test_data.describe()

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>SaleID</th> 
   <th>name</th> 
   <th>regDate</th> 
   <th>model</th> 
   <th>brand</th> 
   <th>bodyType</th> 
   <th>fuelType</th> 
   <th>gearbox</th> 
   <th>power</th> 
   <th>kilometer</th> 
   <th>...</th> 
   <th>v_5</th> 
   <th>v_6</th> 
   <th>v_7</th> 
   <th>v_8</th> 
   <th>v_9</th> 
   <th>v_10</th> 
   <th>v_11</th> 
   <th>v_12</th> 
   <th>v_13</th> 
   <th>v_14</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>count</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>5.000000e+04</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>48587.000000</td> 
   <td>47107.000000</td> 
   <td>48090.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>...</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
   <td>50000.000000</td> 
  </tr> 
  <tr> 
   <td>mean</td> 
   <td>174999.500000</td> 
   <td>68542.223280</td> 
   <td>2.003393e+07</td> 
   <td>46.844520</td> 
   <td>8.056240</td> 
   <td>1.782185</td> 
   <td>0.373405</td> 
   <td>0.224350</td> 
   <td>119.883620</td> 
   <td>12.595580</td> 
   <td>...</td> 
   <td>0.248669</td> 
   <td>0.045021</td> 
   <td>0.122744</td> 
   <td>0.057997</td> 
   <td>0.062000</td> 
   <td>-0.017855</td> 
   <td>-0.013742</td> 
   <td>-0.013554</td> 
   <td>-0.003147</td> 
   <td>0.001516</td> 
  </tr> 
  <tr> 
   <td>std</td> 
   <td>14433.901067</td> 
   <td>61052.808133</td> 
   <td>5.368870e+04</td> 
   <td>49.469548</td> 
   <td>7.819477</td> 
   <td>1.760736</td> 
   <td>0.546442</td> 
   <td>0.417158</td> 
   <td>185.097387</td> 
   <td>3.908979</td> 
   <td>...</td> 
   <td>0.044601</td> 
   <td>0.051766</td> 
   <td>0.195972</td> 
   <td>0.029211</td> 
   <td>0.035653</td> 
   <td>3.747985</td> 
   <td>3.231258</td> 
   <td>2.515962</td> 
   <td>1.286597</td> 
   <td>1.027360</td> 
  </tr> 
  <tr> 
   <td>min</td> 
   <td>150000.000000</td> 
   <td>0.000000</td> 
   <td>1.991000e+07</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.500000</td> 
   <td>...</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>-9.160049</td> 
   <td>-5.411964</td> 
   <td>-8.916949</td> 
   <td>-4.123333</td> 
   <td>-6.112667</td> 
  </tr> 
  <tr> 
   <td>25%</td> 
   <td>162499.750000</td> 
   <td>11203.500000</td> 
   <td>1.999091e+07</td> 
   <td>10.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>75.000000</td> 
   <td>12.500000</td> 
   <td>...</td> 
   <td>0.243762</td> 
   <td>0.000044</td> 
   <td>0.062644</td> 
   <td>0.035084</td> 
   <td>0.033714</td> 
   <td>-3.700121</td> 
   <td>-1.971325</td> 
   <td>-1.876703</td> 
   <td>-1.060428</td> 
   <td>-0.437920</td> 
  </tr> 
  <tr> 
   <td>50%</td> 
   <td>174999.500000</td> 
   <td>52248.500000</td> 
   <td>2.003091e+07</td> 
   <td>29.000000</td> 
   <td>6.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>0.000000</td> 
   <td>109.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.257877</td> 
   <td>0.000815</td> 
   <td>0.095828</td> 
   <td>0.057084</td> 
   <td>0.058764</td> 
   <td>1.613212</td> 
   <td>-0.355843</td> 
   <td>-0.142779</td> 
   <td>-0.035956</td> 
   <td>0.138799</td> 
  </tr> 
  <tr> 
   <td>75%</td> 
   <td>187499.250000</td> 
   <td>118856.500000</td> 
   <td>2.007110e+07</td> 
   <td>65.000000</td> 
   <td>13.000000</td> 
   <td>3.000000</td> 
   <td>1.000000</td> 
   <td>0.000000</td> 
   <td>150.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.265328</td> 
   <td>0.102025</td> 
   <td>0.125438</td> 
   <td>0.079077</td> 
   <td>0.087489</td> 
   <td>2.832708</td> 
   <td>1.262914</td> 
   <td>1.764335</td> 
   <td>0.941469</td> 
   <td>0.681163</td> 
  </tr> 
  <tr> 
   <td>max</td> 
   <td>199999.000000</td> 
   <td>196805.000000</td> 
   <td>2.015121e+07</td> 
   <td>246.000000</td> 
   <td>39.000000</td> 
   <td>7.000000</td> 
   <td>6.000000</td> 
   <td>1.000000</td> 
   <td>20000.000000</td> 
   <td>15.000000</td> 
   <td>...</td> 
   <td>0.291618</td> 
   <td>0.153265</td> 
   <td>1.358813</td> 
   <td>0.156355</td> 
   <td>0.214775</td> 
   <td>12.338872</td> 
   <td>18.856218</td> 
   <td>12.950498</td> 
   <td>5.913273</td> 
   <td>2.624622</td> 
  </tr> 
 </tbody> 
</table>

8 rows × 29 columns

## 2) 通过info()来熟悉数据类型
    Train_data.info()

<class 'pandas.core.frame.DataFrame'>
    RangeIndex: 150000 entries, 0 to 149999
    Data columns (total 31 columns):
    SaleID               150000 non-null int64
    name                 150000 non-null int64
    regDate              150000 non-null int64
    model                149999 non-null float64
    brand                150000 non-null int64
    bodyType             145494 non-null float64
    fuelType             141320 non-null float64
    gearbox              144019 non-null float64
    power                150000 non-null int64
    kilometer            150000 non-null float64
    notRepairedDamage    150000 non-null object
    regionCode           150000 non-null int64
    seller               150000 non-null int64
    offerType            150000 non-null int64
    creatDate            150000 non-null int64
    price                150000 non-null int64
    v_0                  150000 non-null float64
    v_1                  150000 non-null float64
    v_2                  150000 non-null float64
    v_3                  150000 non-null float64
    v_4                  150000 non-null float64
    v_5                  150000 non-null float64
    v_6                  150000 non-null float64
    v_7                  150000 non-null float64
    v_8                  150000 non-null float64
    v_9                  150000 non-null float64
    v_10                 150000 non-null float64
    v_11                 150000 non-null float64
    v_12                 150000 non-null float64
    v_13                 150000 non-null float64
    v_14                 150000 non-null float64
    dtypes: float64(20), int64(10), object(1)
    memory usage: 35.5+ MB

Test_data.info()

<class 'pandas.core.frame.DataFrame'>
    RangeIndex: 50000 entries, 0 to 49999
    Data columns (total 30 columns):
    SaleID               50000 non-null int64
    name                 50000 non-null int64
    regDate              50000 non-null int64
    model                50000 non-null float64
    brand                50000 non-null int64
    bodyType             48587 non-null float64
    fuelType             47107 non-null float64
    gearbox              48090 non-null float64
    power                50000 non-null int64
    kilometer            50000 non-null float64
    notRepairedDamage    50000 non-null object
    regionCode           50000 non-null int64
    seller               50000 non-null int64
    offerType            50000 non-null int64
    creatDate            50000 non-null int64
    v_0                  50000 non-null float64
    v_1                  50000 non-null float64
    v_2                  50000 non-null float64
    v_3                  50000 non-null float64
    v_4                  50000 non-null float64
    v_5                  50000 non-null float64
    v_6                  50000 non-null float64
    v_7                  50000 non-null float64
    v_8                  50000 non-null float64
    v_9                  50000 non-null float64
    v_10                 50000 non-null float64
    v_11                 50000 non-null float64
    v_12                 50000 non-null float64
    v_13                 50000 non-null float64
    v_14                 50000 non-null float64
    dtypes: float64(20), int64(9), object(1)
    memory usage: 11.4+ MB

# 缺失值和异常值分析 #

## 1) 查看每列的存在nan情况
    Train_data.isnull().sum()

SaleID                  0
    name                    0
    regDate                 0
    model                   1
    brand                   0
    bodyType             4506
    fuelType             8680
    gearbox              5981
    power                   0
    kilometer               0
    notRepairedDamage       0
    regionCode              0
    seller                  0
    offerType               0
    creatDate               0
    price                   0
    v_0                     0
    v_1                     0
    v_2                     0
    v_3                     0
    v_4                     0
    v_5                     0
    v_6                     0
    v_7                     0
    v_8                     0
    v_9                     0
    v_10                    0
    v_11                    0
    v_12                    0
    v_13                    0
    v_14                    0
    dtype: int64

Test_data.isnull().sum()

SaleID                  0
    name                    0
    regDate                 0
    model                   0
    brand                   0
    bodyType             1413
    fuelType             2893
    gearbox              1910
    power                   0
    kilometer               0
    notRepairedDamage       0
    regionCode              0
    seller                  0
    offerType               0
    creatDate               0
    v_0                     0
    v_1                     0
    v_2                     0
    v_3                     0
    v_4                     0
    v_5                     0
    v_6                     0
    v_7                     0
    v_8                     0
    v_9                     0
    v_10                    0
    v_11                    0
    v_12                    0
    v_13                    0
    v_14                    0
    dtype: int64

# nan可视化
    missing = Train_data.isnull().sum()
    missing = missing[missing > 0]# 取出大于0的
    missing.sort_values(inplace=True)# 升序排列
    missing.plot.bar()

<matplotlib.axes._subplots.AxesSubplot at 0x7f5add5da890>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Rat0Ogt3-1584885647200)(output\_19\_1.png)\]

*  通过以上两句可以很直观的了解哪些列存在 “nan”, 并可以把nan的个数打印，主要的目的在于 nan存在的个数是否真的很大，如果很小一般选择填充，如果使用lgb等树模型可以直接空缺，让树自己去优化，但如果nan存在的过多、可以考虑删掉

# 可视化看下缺省值
    msno.matrix(Train_data.sample(250))

<matplotlib.axes._subplots.AxesSubplot at 0x7f5adc9c3390>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1W3ADPqM-1584885647201)(output\_21\_1.png)\]

msno.bar(Train_data.sample(1000))

<matplotlib.axes._subplots.AxesSubplot at 0x7f5adc86b090>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-yUUKgDZP-1584885647202)(output\_22\_1.png)\]

# 可视化看下缺省值
    msno.matrix(Test_data.sample(250))

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ada277450>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lBPoeuSV-1584885647203)(output\_23\_1.png)\]

msno.bar(Test_data.sample(1000))

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ada2ca050>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-KGNRJexG-1584885647204)(output\_24\_1.png)\]

测试集的缺省和训练集的差不多情况, 可视化有四列有缺省，notRepairedDamage缺省得最多

# 异常值检测 #

Train_data.info()

可以发现除了notRepairedDamage 为object类型其他都为数字 这里我们把他的几个不同的值都进行显示就知道了

Train_data['notRepairedDamage'].value_counts()

0.0    111361
    -       24324
    1.0     14315
    Name: notRepairedDamage, dtype: int64

*  可以看出来‘ - ’也为空缺值，因为很多模型对nan有直接的处理，这里我们先不做处理，先替换成nan

Train_data['notRepairedDamage'].replace('-', np.nan, inplace=True)

Train_data['notRepairedDamage'].value_counts()

0.0    111361
    1.0     14315
    Name: notRepairedDamage, dtype: int64

Train_data.isnull().sum()

SaleID                   0
    name                     0
    regDate                  0
    model                    1
    brand                    0
    bodyType              4506
    fuelType              8680
    gearbox               5981
    power                    0
    kilometer                0
    notRepairedDamage    24324
    regionCode               0
    seller                   0
    offerType                0
    creatDate                0
    price                    0
    v_0                      0
    v_1                      0
    v_2                      0
    v_3                      0
    v_4                      0
    v_5                      0
    v_6                      0
    v_7                      0
    v_8                      0
    v_9                      0
    v_10                     0
    v_11                     0
    v_12                     0
    v_13                     0
    v_14                     0
    dtype: int64

# 特征倾斜检测 #

Train_data["seller"].value_counts()

0    149999
    1         1
    Name: seller, dtype: int64

Train_data["offerType"].value_counts()

0    150000
    Name: offerType, dtype: int64

del Train_data["seller"]
    del Train_data["offerType"]
    del Test_data["seller"]
    del Test_data["offerType"]

# 预测值分布 #

Train_data['price']

0         1850
    1         3600
    2         6222
    3         2400
    4         5200
              ... 
    149995    5900
    149996    9500
    149997    7500
    149998    4999
    149999    4700
    Name: price, Length: 150000, dtype: int64

Train_data['price'].value_counts()

500      2337
    1500     2158
    1200     1922
    1000     1850
    2500     1821
             ... 
    25321       1
    8886        1
    8801        1
    37920       1
    8188        1
    Name: price, Length: 3763, dtype: int64

## 1) 总体分布概况（无界约翰逊分布等）
    import scipy.stats as st
    y = Train_data['price']
    plt.figure(1); plt.title('Johnson SU')
    sns.distplot(y, kde=False, fit=st.johnsonsu)
    plt.figure(2); plt.title('Normal')
    sns.distplot(y, kde=False, fit=st.norm)
    plt.figure(3); plt.title('Log Normal')
    sns.distplot(y, kde=False, fit=st.lognorm)

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ad9e69110>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zC6ld8ar-1584885647204)(output\_41\_1.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nLEqA09J-1584885647205)(output\_41\_2.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-kS7jNVAq-1584885647206)(output\_41\_3.png)\]

*  价格不服从正态分布，所以在进行回归之前，它必须进行转换。虽然对数变换做得很好，但最佳拟合是无界约翰逊分布

## 2) 查看skewness and kurtosis
    sns.distplot(Train_data['price']);
    print("Skewness: %f" % Train_data['price'].skew())
    print("Kurtosis: %f" % Train_data['price'].kurt())

Skewness: 3.346487
    Kurtosis: 18.995183

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-bEUJjMK9-1584885647206)(output\_43\_1.png)\]

Train_data.skew(), Train_data.kurt()

(SaleID               6.017846e-17
     name                 5.576058e-01
     regDate              2.849508e-02
     model                1.484388e+00
     brand                1.150760e+00
     bodyType             9.915299e-01
     fuelType             1.595486e+00
     gearbox              1.317514e+00
     power                6.586318e+01
     kilometer           -1.525921e+00
     notRepairedDamage    2.430640e+00
     regionCode           6.888812e-01
     creatDate           -7.901331e+01
     price                3.346487e+00
     v_0                 -1.316712e+00
     v_1                  3.594543e-01
     v_2                  4.842556e+00
     v_3                  1.062920e-01
     v_4                  3.679890e-01
     v_5                 -4.737094e+00
     v_6                  3.680730e-01
     v_7                  5.130233e+00
     v_8                  2.046133e-01
     v_9                  4.195007e-01
     v_10                 2.522046e-02
     v_11                 3.029146e+00
     v_12                 3.653576e-01
     v_13                 2.679152e-01
     v_14                -1.186355e+00
     dtype: float64, SaleID                 -1.200000
     name                   -1.039945
     regDate                -0.697308
     model                   1.740483
     brand                   1.076201
     bodyType                0.206937
     fuelType                5.880049
     gearbox                -0.264161
     power                5733.451054
     kilometer               1.141934
     notRepairedDamage       3.908072
     regionCode             -0.340832
     creatDate            6881.080328
     price                  18.995183
     v_0                     3.993841
     v_1                    -1.753017
     v_2                    23.860591
     v_3                    -0.418006
     v_4                    -0.197295
     v_5                    22.934081
     v_6                    -1.742567
     v_7                    25.845489
     v_8                    -0.636225
     v_9                    -0.321491
     v_10                   -0.577935
     v_11                   12.568731
     v_12                    0.268937
     v_13                   -0.438274
     v_14                    2.393526
     dtype: float64)

sns.distplot(Train_data.skew(),color='blue',axlabel ='Skewness')

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ad8526510>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-5dij5RqL-1584885647207)(output\_45\_1.png)\]

sns.distplot(Train_data.kurt(),color='orange',axlabel ='Kurtness')

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ad83da590>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-U89djmGc-1584885647208)(output\_46\_1.png)\]

skew、kurt说明参考https://www.cnblogs.com/wyy1480/p/10474046.html

## 3) 查看预测值的具体频数
    plt.hist(Train_data['price'], orientation = 'vertical',histtype = 'bar', color ='red')
    plt.show()

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-wvP15hTK-1584885647208)(output\_48\_0.png)\]

查看频数, 大于20000得值极少，其实这里也可以把这些当作特殊得值（异常值）直接用填充或者删掉，再前面进行

# log变换 z之后的分布较均匀，可以进行log变换进行预测，这也是预测问题常用的trick
    plt.hist(np.log(Train_data['price']), orientation = 'vertical',histtype = 'bar', color ='red') 
    plt.show()

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-mSQDYK0r-1584885647209)(output\_50\_0.png)\]

# 数据类型分析 #

**列**

*  name - 汽车编码
 *  regDate - 汽车注册时间
 *  model - 车型编码
 *  brand - 品牌
 *  bodyType - 车身类型
 *  fuelType - 燃油类型
 *  gearbox - 变速箱
 *  power - 汽车功率
 *  kilometer - 汽车行驶公里
 *  notRepairedDamage - 汽车有尚未修复的损坏
 *  regionCode - 看车地区编码
 *  seller - 销售方 【以删】
 *  offerType - 报价类型 【以删】
 *  creatDate - 广告发布时间
 *  price - 汽车价格
 *  v\_0’, ‘v\_1’, ‘v\_2’, ‘v\_3’, ‘v\_4’, ‘v\_5’, ‘v\_6’, ‘v\_7’, ‘v\_8’, ‘v\_9’, ‘v\_10’, ‘v\_11’, ‘v\_12’, ‘v\_13’,‘v\_14’（根据汽车的评论、标签等大量信息得到的embedding向量）【人工构造 匿名特征】

# 分离label即预测值
    Y_train = Train_data['price']

# 这个区别方式适用于没有直接label coding的数据
    # 这里不适用，需要人为根据实际含义来区分
    # 数字特征
    # numeric_features = Train_data.select_dtypes(include=[np.number])
    # numeric_features.columns
    # # 类型特征
    # categorical_features = Train_data.select_dtypes(include=[np.object])
    # categorical_features.columns

#数字特征和分类特征
    numeric_features = ['power', 'kilometer', 'v_0', 'v_1', 'v_2', 'v_3', 'v_4', 'v_5', 'v_6', 'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12', 'v_13','v_14' ]
    
    categorical_features = ['name', 'model', 'brand', 'bodyType', 'fuelType', 'gearbox', 'notRepairedDamage', 'regionCode',]

# 特征nunique分布
    for cat_fea in categorical_features:
        print(cat_fea + "的特征分布如下：")
        print("{}特征有个{}不同的值".format(cat_fea, Train_data[cat_fea].nunique()))
        print(Train_data[cat_fea].value_counts())

name的特征分布如下：
    name特征有个99662不同的值
    708       282
    387       282
    55        280
    1541      263
    203       233
             ... 
    5074        1
    7123        1
    11221       1
    13270       1
    174485      1
    Name: name, Length: 99662, dtype: int64
    model的特征分布如下：
    model特征有个248不同的值
    0.0      11762
    19.0      9573
    4.0       8445
    1.0       6038
    29.0      5186
             ...  
    245.0        2
    209.0        2
    240.0        2
    242.0        2
    247.0        1
    Name: model, Length: 248, dtype: int64
    brand的特征分布如下：
    brand特征有个40不同的值
    0     31480
    4     16737
    14    16089
    10    14249
    1     13794
    6     10217
    9      7306
    5      4665
    13     3817
    11     2945
    3      2461
    7      2361
    16     2223
    8      2077
    25     2064
    27     2053
    21     1547
    15     1458
    19     1388
    20     1236
    12     1109
    22     1085
    26      966
    30      940
    17      913
    24      772
    28      649
    32      592
    29      406
    37      333
    2       321
    31      318
    18      316
    36      228
    34      227
    33      218
    23      186
    35      180
    38       65
    39        9
    Name: brand, dtype: int64
    bodyType的特征分布如下：
    bodyType特征有个8不同的值
    0.0    41420
    1.0    35272
    2.0    30324
    3.0    13491
    4.0     9609
    5.0     7607
    6.0     6482
    7.0     1289
    Name: bodyType, dtype: int64
    fuelType的特征分布如下：
    fuelType特征有个7不同的值
    0.0    91656
    1.0    46991
    2.0     2212
    3.0      262
    4.0      118
    5.0       45
    6.0       36
    Name: fuelType, dtype: int64
    gearbox的特征分布如下：
    gearbox特征有个2不同的值
    0.0    111623
    1.0     32396
    Name: gearbox, dtype: int64
    notRepairedDamage的特征分布如下：
    notRepairedDamage特征有个2不同的值
    0.0    111361
    1.0     14315
    Name: notRepairedDamage, dtype: int64
    regionCode的特征分布如下：
    regionCode特征有个7905不同的值
    419     369
    764     258
    125     137
    176     136
    462     134
           ... 
    6414      1
    7063      1
    4239      1
    5931      1
    7267      1
    Name: regionCode, Length: 7905, dtype: int64

# 特征nunique分布
    for cat_fea in categorical_features:
        print(cat_fea + "的特征分布如下：")
        print("{}特征有个{}不同的值".format(cat_fea, Test_data[cat_fea].nunique()))
        print(Test_data[cat_fea].value_counts())

name的特征分布如下：
    name特征有个37453不同的值
    55       97
    708      96
    387      95
    1541     88
    713      74
             ..
    22270     1
    89855     1
    42752     1
    48899     1
    11808     1
    Name: name, Length: 37453, dtype: int64
    model的特征分布如下：
    model特征有个247不同的值
    0.0      3896
    19.0     3245
    4.0      3007
    1.0      1981
    29.0     1742
             ... 
    242.0       1
    240.0       1
    244.0       1
    243.0       1
    246.0       1
    Name: model, Length: 247, dtype: int64
    brand的特征分布如下：
    brand特征有个40不同的值
    0     10348
    4      5763
    14     5314
    10     4766
    1      4532
    6      3502
    9      2423
    5      1569
    13     1245
    11      919
    7       795
    3       773
    16      771
    8       704
    25      695
    27      650
    21      544
    15      511
    20      450
    19      450
    12      389
    22      363
    30      324
    17      317
    26      303
    24      268
    28      225
    32      193
    29      117
    31      115
    18      106
    2       104
    37       92
    34       77
    33       76
    36       67
    23       62
    35       53
    38       23
    39        2
    Name: brand, dtype: int64
    bodyType的特征分布如下：
    bodyType特征有个8不同的值
    0.0    13985
    1.0    11882
    2.0     9900
    3.0     4433
    4.0     3303
    5.0     2537
    6.0     2116
    7.0      431
    Name: bodyType, dtype: int64
    fuelType的特征分布如下：
    fuelType特征有个7不同的值
    0.0    30656
    1.0    15544
    2.0      774
    3.0       72
    4.0       37
    6.0       14
    5.0       10
    Name: fuelType, dtype: int64
    gearbox的特征分布如下：
    gearbox特征有个2不同的值
    0.0    37301
    1.0    10789
    Name: gearbox, dtype: int64
    notRepairedDamage的特征分布如下：
    notRepairedDamage特征有个3不同的值
    0.0    37249
    -       8031
    1.0     4720
    Name: notRepairedDamage, dtype: int64
    regionCode的特征分布如下：
    regionCode特征有个6971不同的值
    419     146
    764      78
    188      52
    125      51
    759      51
           ... 
    7753      1
    7463      1
    7230      1
    826       1
    112       1
    Name: regionCode, Length: 6971, dtype: int64

# 数字特征分析 #

numeric_features.append('price')

numeric_features

['power',
     'kilometer',
     'v_0',
     'v_1',
     'v_2',
     'v_3',
     'v_4',
     'v_5',
     'v_6',
     'v_7',
     'v_8',
     'v_9',
     'v_10',
     'v_11',
     'v_12',
     'v_13',
     'v_14',
     'price']

Train_data.head()

5 rows × 29 columns

## 1) 相关性分析
    price_numeric = Train_data[numeric_features]
    correlation = price_numeric.corr()
    print(correlation['price'].sort_values(ascending = False),'\n')

price        1.000000
    v_12         0.692823
    v_8          0.685798
    v_0          0.628397
    power        0.219834
    v_5          0.164317
    v_2          0.085322
    v_6          0.068970
    v_1          0.060914
    v_14         0.035911
    v_13        -0.013993
    v_7         -0.053024
    v_4         -0.147085
    v_9         -0.206205
    v_10        -0.246175
    v_11        -0.275320
    kilometer   -0.440519
    v_3         -0.730946
    Name: price, dtype: float64

f , ax = plt.subplots(figsize = (7, 7))
    
    plt.title('Correlation of Numeric Features with Price',y=1,size=16)
    
    sns.heatmap(correlation,square = True,  vmax=0.8)

<matplotlib.axes._subplots.AxesSubplot at 0x7f5ad826aa10>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-bvDY9ToA-1584885647210)(output\_63\_1.png)\]

del price_numeric['price']

## 2) 查看几个特征得 偏度和峰值
    for col in numeric_features:
        print('{:15}'.format(col), 
              'Skewness: {:05.2f}'.format(Train_data[col].skew()) , 
              ' ' ,
              'Kurtosis: {:06.2f}'.format(Train_data[col].kurt())  
             )

power           Skewness: 65.86     Kurtosis: 5733.45
    kilometer       Skewness: -1.53     Kurtosis: 001.14
    v_0             Skewness: -1.32     Kurtosis: 003.99
    v_1             Skewness: 00.36     Kurtosis: -01.75
    v_2             Skewness: 04.84     Kurtosis: 023.86
    v_3             Skewness: 00.11     Kurtosis: -00.42
    v_4             Skewness: 00.37     Kurtosis: -00.20
    v_5             Skewness: -4.74     Kurtosis: 022.93
    v_6             Skewness: 00.37     Kurtosis: -01.74
    v_7             Skewness: 05.13     Kurtosis: 025.85
    v_8             Skewness: 00.20     Kurtosis: -00.64
    v_9             Skewness: 00.42     Kurtosis: -00.32
    v_10            Skewness: 00.03     Kurtosis: -00.58
    v_11            Skewness: 03.03     Kurtosis: 012.57
    v_12            Skewness: 00.37     Kurtosis: 000.27
    v_13            Skewness: 00.27     Kurtosis: -00.44
    v_14            Skewness: -1.19     Kurtosis: 002.39
    price           Skewness: 03.35     Kurtosis: 019.00

## 3) 每个数字特征得分布可视化
    f = pd.melt(Train_data, value_vars=numeric_features)
    g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False)
    g = g.map(sns.distplot, "value")

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-by4pagHn-1584885647210)(output\_66\_0.png)\]

# 可以看出匿名特征相对分布均匀 #

## 4) 数字特征相互之间的关系可视化
    sns.set()
    columns = ['price', 'v_12', 'v_8' , 'v_0', 'power', 'v_5',  'v_2', 'v_6', 'v_1', 'v_14']
    sns.pairplot(Train_data[columns],size = 2 ,kind ='scatter',diag_kind='kde')
    plt.show()

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Ylf7L29K-1584885647211)(output\_68\_0.png)\]

Train_data.columns

Index(['SaleID', 'name', 'regDate', 'model', 'brand', 'bodyType', 'fuelType',
           'gearbox', 'power', 'kilometer', 'notRepairedDamage', 'regionCode',
           'creatDate', 'price', 'v_0', 'v_1', 'v_2', 'v_3', 'v_4', 'v_5', 'v_6',
           'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12', 'v_13', 'v_14'],
          dtype='object')

#### 此处是多变量之间的关系可视化，可视化更多学习可参考很不错的文章 https://www.jianshu.com/p/6e18d21a4cad ####

## 5) 多变量互相回归关系可视化
    fig, ((ax1, ax2), (ax3, ax4), (ax5, ax6), (ax7, ax8), (ax9, ax10)) = plt.subplots(nrows=5, ncols=2, figsize=(24, 20))
    # ['v_12', 'v_8' , 'v_0', 'power', 'v_5', 'v_2', 'v_6', 'v_1', 'v_14']
    v_12_scatter_plot = pd.concat([Y_train,Train_data['v_12']],axis = 1)
    sns.regplot(x='v_12',y = 'price', data = v_12_scatter_plot,scatter= True, fit_reg=True, ax=ax1)
    
    v_8_scatter_plot = pd.concat([Y_train,Train_data['v_8']],axis = 1)
    sns.regplot(x='v_8',y = 'price',data = v_8_scatter_plot,scatter= True, fit_reg=True, ax=ax2)
    
    v_0_scatter_plot = pd.concat([Y_train,Train_data['v_0']],axis = 1)
    sns.regplot(x='v_0',y = 'price',data = v_0_scatter_plot,scatter= True, fit_reg=True, ax=ax3)
    
    power_scatter_plot = pd.concat([Y_train,Train_data['power']],axis = 1)
    sns.regplot(x='power',y = 'price',data = power_scatter_plot,scatter= True, fit_reg=True, ax=ax4)
    
    v_5_scatter_plot = pd.concat([Y_train,Train_data['v_5']],axis = 1)
    sns.regplot(x='v_5',y = 'price',data = v_5_scatter_plot,scatter= True, fit_reg=True, ax=ax5)
    
    v_2_scatter_plot = pd.concat([Y_train,Train_data['v_2']],axis = 1)
    sns.regplot(x='v_2',y = 'price',data = v_2_scatter_plot,scatter= True, fit_reg=True, ax=ax6)
    
    v_6_scatter_plot = pd.concat([Y_train,Train_data['v_6']],axis = 1)
    sns.regplot(x='v_6',y = 'price',data = v_6_scatter_plot,scatter= True, fit_reg=True, ax=ax7)
    
    v_1_scatter_plot = pd.concat([Y_train,Train_data['v_1']],axis = 1)
    sns.regplot(x='v_1',y = 'price',data = v_1_scatter_plot,scatter= True, fit_reg=True, ax=ax8)
    
    v_14_scatter_plot = pd.concat([Y_train,Train_data['v_14']],axis = 1)
    sns.regplot(x='v_14',y = 'price',data = v_14_scatter_plot,scatter= True, fit_reg=True, ax=ax9)
    
    v_13_scatter_plot = pd.concat([Y_train,Train_data['v_13']],axis = 1)
    sns.regplot(x='v_13',y = 'price',data = v_13_scatter_plot,scatter= True, fit_reg=True, ax=ax10)

<matplotlib.axes._subplots.AxesSubplot at 0x7f5acf0c9ad0>

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zWyh16N5-1584885647212)(output\_71\_1.png)\]

### 类别特征分析 ###

## 1) unique分布
    for fea in categorical_features:
        print(Train_data[fea].nunique())

99662
    248
    40
    8
    7
    2
    2
    7905

categorical_features

['name',
     'model',
     'brand',
     'bodyType',
     'fuelType',
     'gearbox',
     'notRepairedDamage',
     'regionCode']

## 2) 类别特征箱形图可视化
    
    # 因为 name和 regionCode的类别太稀疏了，这里我们把不稀疏的几类画一下
    categorical_features = ['model',
     'brand',
     'bodyType',
     'fuelType',
     'gearbox',
     'notRepairedDamage']
    for c in categorical_features:
        Train_data[c] = Train_data[c].astype('category')
        if Train_data[c].isnull().any():#df.isnull().any()则会判断哪些”列”存在缺失值
            Train_data[c] = Train_data[c].cat.add_categories(['MISSING'])#添加新的类别
            Train_data[c] = Train_data[c].fillna('MISSING')
    
    def boxplot(x, y, **kwargs):
        sns.boxplot(x=x, y=y)
        x=plt.xticks(rotation=90)
    
    f = pd.melt(Train_data, id_vars=['price'], value_vars=categorical_features)
    g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False, size=5)
    g = g.map(boxplot, "value", "price")# 这个函数不知道什么意思

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lVfH0HBX-1584885647212)(output\_75\_0.png)\]

Train_data.columns

## 3) 类别特征的小提琴图可视化
    catg_list = categorical_features
    target = 'price'
    for catg in catg_list :
        sns.violinplot(x=catg, y=target, data=Train_data)
        plt.show()

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-XhN6yWh9-1584885647213)(output\_77\_0.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-h8kBD1q7-1584885647214)(output\_77\_1.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-37IwCCm6-1584885647214)(output\_77\_2.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0Fy3KPHQ-1584885647215)(output\_77\_3.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pc7W07IG-1584885647216)(output\_77\_4.png)\]

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1MuF3E3z-1584885647216)(output\_77\_5.png)\]

categorical_features = ['model',
     'brand',
     'bodyType',
     'fuelType',
     'gearbox',
     'notRepairedDamage']

## 4) 类别特征的柱形图可视化
    def bar_plot(x, y, **kwargs):
        sns.barplot(x=x, y=y)
        x=plt.xticks(rotation=90)
    
    f = pd.melt(Train_data, id_vars=['price'], value_vars=categorical_features)
    g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False, size=5)
    g = g.map(bar_plot, "value", "price")

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-4G4l2X05-1584885647217)(output\_79\_0.png)\]

## 5) 类别特征的每个类别频数可视化(count_plot)
    def count_plot(x,  **kwargs):
        sns.countplot(x=x)
        x=plt.xticks(rotation=90)
    
    f = pd.melt(Train_data,  value_vars=categorical_features)
    g = sns.FacetGrid(f, col="variable",  col_wrap=2, sharex=False, sharey=False, size=5)
    g = g.map(count_plot, "value")

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-M4DZ4jf8-1584885647217)(output\_80\_0.png)\]

# 用pandas\_profiling生成数据报告 #

用pandas\_profiling生成一个较为全面的可视化和数据报告(较为简单、方便) 最终打开html文件即可

!pip install pandas_profiling
    import pandas_profiling

Collecting pandas_profiling
    [?25l  Downloading https://files.pythonhosted.org/packages/e8/b0/bd5e3aaf37302fbe581b6947dc5ec1cda02a0ffe50fc823123def73e4d7a/pandas-profiling-2.5.0.tar.gz (192kB)
    [K     |████████████████████████████████| 194kB 498kB/s eta 0:00:01
    [?25hRequirement already satisfied: numpy>=1.16.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas_profiling) (1.17.2)
    Collecting scipy>=1.4.1 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/dd/82/c1fe128f3526b128cfd185580ba40d01371c5d299fcf7f77968e22dfcc2e/scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1MB)
    [K     |████████████████████████████████| 26.1MB 9.8MB/s eta 0:00:011    |████████████                    | 9.8MB 12.8MB/s eta 0:00:02     |█████████████████████▎          | 17.3MB 10.2MB/s eta 0:00:01
    [?25hCollecting pandas==0.25.3 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/63/e0/a1b39cdcb2c391f087a1538bc8a6d62a82d0439693192aef541d7b123769/pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4MB)
    [K     |████████████████████████████████| 10.4MB 11.2MB/s eta 0:00:01
    [?25hRequirement already satisfied: matplotlib>=3.0.3 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas_profiling) (3.1.1)
    Collecting confuse==1.0.0 (from pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/4c/6f/90e860cba937c174d8b3775729ccc6377eb91f52ad4eeb008e7252a3646d/confuse-1.0.0.tar.gz
    Collecting jinja2==2.11.1 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/27/24/4f35961e5c669e96f6559760042a55b9bcfcdb82b9bdb3c8753dbe042e35/Jinja2-2.11.1-py2.py3-none-any.whl (126kB)
    [K     |████████████████████████████████| 133kB 12.2MB/s eta 0:00:01
    [?25hCollecting visions==0.2.2 (from pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/07/4a/ab37f8bafda516b66c4f475b221a6c170097c0db203750a4aafb01023339/visions-0.2.2.tar.gz
    Collecting htmlmin==0.1.12 (from pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/b3/e7/fcd59e12169de19f0131ff2812077f964c6b960e7c09804d30a7bf2ab461/htmlmin-0.1.12.tar.gz
    Requirement already satisfied: missingno==0.4.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas_profiling) (0.4.2)
    Collecting phik==0.9.9 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/03/cf/b8cef2778104dc5d319f36dd836efaceb07a037cbf63f27c966b5a193ce9/phik-0.9.9-py3-none-any.whl (607kB)
    [K     |████████████████████████████████| 614kB 11.4MB/s eta 0:00:01
    [?25hCollecting astropy>=3.2.3 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/d5/50/5195c71d9aed4b5673a820724dada56e21f65f717c68a0e29183b4e12375/astropy-4.0-cp37-cp37m-manylinux1_x86_64.whl (6.5MB)
    [K     |████████████████████████████████| 6.5MB 2.2MB/s eta 0:00:011
    [?25hCollecting tangled-up-in-unicode==0.0.3 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/43/fc/e3c970c5007b405827a4623e70fc4eb966ee49bc1edbd56c33e85e1c6534/tangled_up_in_unicode-0.0.3.tar.gz (1.5MB)
    [K     |████████████████████████████████| 1.5MB 8.0MB/s eta 0:00:01
    [?25hCollecting tqdm==4.42.0 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/cc/2e/4307206db63f05ed37e21d4c0d843d0fbcacd62479f8ce99ba0f2c0875e0/tqdm-4.42.0-py2.py3-none-any.whl (59kB)
    [K     |████████████████████████████████| 61kB 5.8MB/s eta 0:00:01
    [?25hCollecting kaggle==1.5.6 (from pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/62/ab/bb20f9b9e24f9a6250f95a432f8d9a7d745f8d24039d7a5a6eaadb7783ba/kaggle-1.5.6.tar.gz (58kB)
    [K     |████████████████████████████████| 61kB 5.8MB/s  eta 0:00:01
    [?25hRequirement already satisfied: ipywidgets==7.5.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas_profiling) (7.5.1)
    Requirement already satisfied: requests==2.22.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas_profiling) (2.22.0)
    Requirement already satisfied: python-dateutil>=2.6.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas==0.25.3->pandas_profiling) (2.8.0)
    Requirement already satisfied: pytz>=2017.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from pandas==0.25.3->pandas_profiling) (2019.3)
    Requirement already satisfied: cycler>=0.10 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib>=3.0.3->pandas_profiling) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib>=3.0.3->pandas_profiling) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from matplotlib>=3.0.3->pandas_profiling) (2.4.2)
    Requirement already satisfied: pyyaml in /home/ach/anaconda3/lib/python3.7/site-packages (from confuse==1.0.0->pandas_profiling) (5.1.2)
    Requirement already satisfied: MarkupSafe>=0.23 in /home/ach/anaconda3/lib/python3.7/site-packages (from jinja2==2.11.1->pandas_profiling) (1.1.1)
    Requirement already satisfied: networkx in /home/ach/anaconda3/lib/python3.7/site-packages (from visions==0.2.2->pandas_profiling) (2.4)
    Collecting attr (from visions==0.2.2->pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/de/be/ddc7f84d4e087144472a38a373d3e319f51a6faf6e5fc1ae897173675f21/attr-0.3.1.tar.gz
    Requirement already satisfied: seaborn in /home/ach/anaconda3/lib/python3.7/site-packages (from missingno==0.4.2->pandas_profiling) (0.9.0)
    Requirement already satisfied: pytest>=4.0.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from phik==0.9.9->pandas_profiling) (5.2.1)
    Requirement already satisfied: jupyter-client>=5.2.3 in /home/ach/anaconda3/lib/python3.7/site-packages (from phik==0.9.9->pandas_profiling) (5.3.3)
    Collecting pytest-pylint>=0.13.0 (from phik==0.9.9->pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/31/ef/e848f832a596a8a40b32d8aa169788b4df167c2d6a5960c01e83a30ebaa7/pytest_pylint-0.15.1-py3-none-any.whl
    Requirement already satisfied: numba>=0.38.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from phik==0.9.9->pandas_profiling) (0.45.1)
    Requirement already satisfied: nbconvert>=5.3.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from phik==0.9.9->pandas_profiling) (5.6.0)
    Collecting joblib>=0.14.1 (from phik==0.9.9->pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/28/5c/cf6a2b65a321c4a209efcdf64c2689efae2cb62661f8f6f4bb28547cf1bf/joblib-0.14.1-py2.py3-none-any.whl (294kB)
    [K     |████████████████████████████████| 296kB 13.4MB/s eta 0:00:01
    [?25hRequirement already satisfied: urllib3<1.25,>=1.21.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from kaggle==1.5.6->pandas_profiling) (1.24.2)
    Requirement already satisfied: six>=1.10 in /home/ach/anaconda3/lib/python3.7/site-packages (from kaggle==1.5.6->pandas_profiling) (1.12.0)
    Requirement already satisfied: certifi in /home/ach/anaconda3/lib/python3.7/site-packages (from kaggle==1.5.6->pandas_profiling) (2019.11.28)
    Collecting python-slugify (from kaggle==1.5.6->pandas_profiling)
      Downloading https://files.pythonhosted.org/packages/92/5f/7b84a0bba8a0fdd50c046f8b57dcf179dc16237ad33446079b7c484de04c/python-slugify-4.0.0.tar.gz
    Requirement already satisfied: nbformat>=4.2.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipywidgets==7.5.1->pandas_profiling) (4.4.0)
    Requirement already satisfied: ipykernel>=4.5.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipywidgets==7.5.1->pandas_profiling) (5.1.2)
    Requirement already satisfied: ipython>=4.0.0; python_version >= "3.3" in /home/ach/anaconda3/lib/python3.7/site-packages (from ipywidgets==7.5.1->pandas_profiling) (7.8.0)
    Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipywidgets==7.5.1->pandas_profiling) (3.5.1)
    Requirement already satisfied: traitlets>=4.3.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipywidgets==7.5.1->pandas_profiling) (4.3.3)
    Requirement already satisfied: idna<2.9,>=2.5 in /home/ach/anaconda3/lib/python3.7/site-packages (from requests==2.22.0->pandas_profiling) (2.8)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from requests==2.22.0->pandas_profiling) (3.0.4)
    Requirement already satisfied: setuptools in /home/ach/anaconda3/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib>=3.0.3->pandas_profiling) (41.4.0)
    Requirement already satisfied: decorator>=4.3.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from networkx->visions==0.2.2->pandas_profiling) (4.4.0)
    Requirement already satisfied: py>=1.5.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (1.8.0)
    Requirement already satisfied: packaging in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (19.2)
    Requirement already satisfied: attrs>=17.4.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (19.2.0)
    Requirement already satisfied: more-itertools>=4.0.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (7.2.0)
    Requirement already satisfied: atomicwrites>=1.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (1.3.0)
    Requirement already satisfied: pluggy<1.0,>=0.12 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (0.13.0)
    Requirement already satisfied: wcwidth in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (0.1.7)
    Requirement already satisfied: importlib-metadata>=0.12 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest>=4.0.2->phik==0.9.9->pandas_profiling) (0.23)
    Requirement already satisfied: pyzmq>=13 in /home/ach/anaconda3/lib/python3.7/site-packages (from jupyter-client>=5.2.3->phik==0.9.9->pandas_profiling) (18.1.0)
    Requirement already satisfied: jupyter-core in /home/ach/anaconda3/lib/python3.7/site-packages (from jupyter-client>=5.2.3->phik==0.9.9->pandas_profiling) (4.5.0)
    Requirement already satisfied: tornado>=4.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from jupyter-client>=5.2.3->phik==0.9.9->pandas_profiling) (6.0.3)
    Requirement already satisfied: pylint>=2.0.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (2.4.2)
    Requirement already satisfied: llvmlite>=0.29.0dev0 in /home/ach/anaconda3/lib/python3.7/site-packages (from numba>=0.38.1->phik==0.9.9->pandas_profiling) (0.29.0)
    Requirement already satisfied: pandocfilters>=1.4.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (1.4.2)
    Requirement already satisfied: bleach in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (3.1.0)
    Requirement already satisfied: entrypoints>=0.2.2 in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (0.3)
    Requirement already satisfied: pygments in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (2.4.2)
    Requirement already satisfied: defusedxml in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (0.6.0)
    Requirement already satisfied: testpath in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (0.4.2)
    Requirement already satisfied: mistune<2,>=0.8.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (0.8.4)
    Collecting text-unidecode>=1.3 (from python-slugify->kaggle==1.5.6->pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/a6/a5/c0b6468d3824fe3fde30dbb5e1f687b291608f9473681bbf7dabbf5a87d7/text_unidecode-1.3-py2.py3-none-any.whl (78kB)
    [K     |████████████████████████████████| 81kB 7.7MB/s  eta 0:00:01
    [?25hRequirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/ach/anaconda3/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets==7.5.1->pandas_profiling) (3.0.2)
    Requirement already satisfied: ipython-genutils in /home/ach/anaconda3/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets==7.5.1->pandas_profiling) (0.2.0)
    Requirement already satisfied: backcall in /home/ach/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (0.1.0)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /home/ach/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (4.7.0)
    Requirement already satisfied: jedi>=0.10 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (0.15.1)
    Requirement already satisfied: pickleshare in /home/ach/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (0.7.5)
    Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (2.0.10)
    Requirement already satisfied: notebook>=4.4.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets==7.5.1->pandas_profiling) (6.0.1)
    Requirement already satisfied: zipp>=0.5 in /home/ach/anaconda3/lib/python3.7/site-packages (from importlib-metadata>=0.12->pytest>=4.0.2->phik==0.9.9->pandas_profiling) (0.6.0)
    Requirement already satisfied: astroid<2.4,>=2.3.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (2.3.1)
    Requirement already satisfied: isort<5,>=4.2.5 in /home/ach/anaconda3/lib/python3.7/site-packages (from pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (4.3.21)
    Requirement already satisfied: mccabe<0.7,>=0.6 in /home/ach/anaconda3/lib/python3.7/site-packages (from pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (0.6.1)
    Requirement already satisfied: webencodings in /home/ach/anaconda3/lib/python3.7/site-packages (from bleach->nbconvert>=5.3.1->phik==0.9.9->pandas_profiling) (0.5.1)
    Requirement already satisfied: pyrsistent>=0.14.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets==7.5.1->pandas_profiling) (0.15.4)
    Requirement already satisfied: ptyprocess>=0.5 in /home/ach/anaconda3/lib/python3.7/site-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (0.6.0)
    Requirement already satisfied: parso>=0.5.0 in /home/ach/anaconda3/lib/python3.7/site-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= "3.3"->ipywidgets==7.5.1->pandas_profiling) (0.5.1)
    Requirement already satisfied: Send2Trash in /home/ach/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets==7.5.1->pandas_profiling) (1.5.0)
    Requirement already satisfied: prometheus-client in /home/ach/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets==7.5.1->pandas_profiling) (0.7.1)
    Requirement already satisfied: terminado>=0.8.1 in /home/ach/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets==7.5.1->pandas_profiling) (0.8.2)
    Requirement already satisfied: wrapt==1.11.* in /home/ach/anaconda3/lib/python3.7/site-packages (from astroid<2.4,>=2.3.0->pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (1.11.2)
    Requirement already satisfied: lazy-object-proxy==1.4.* in /home/ach/anaconda3/lib/python3.7/site-packages (from astroid<2.4,>=2.3.0->pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling) (1.4.2)
    Collecting typed-ast<1.5,>=1.4.0; implementation_name == "cpython" and python_version < "3.8" (from astroid<2.4,>=2.3.0->pylint>=2.0.0->pytest-pylint>=0.13.0->phik==0.9.9->pandas_profiling)
    [?25l  Downloading https://files.pythonhosted.org/packages/5d/10/0c1e8aa723a2b0c4032e048d8e511df82c8a1262f0e1df5e4c54eb2613e9/typed_ast-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (737kB)
    [K     |████████████████████████████████| 747kB 10.8MB/s eta 0:00:01
    [?25hBuilding wheels for collected packages: pandas-profiling, confuse, visions, htmlmin, tangled-up-in-unicode, kaggle, attr, python-slugify
      Building wheel for pandas-profiling (setup.py) ... [?25ldone
    [?25h  Created wheel for pandas-profiling: filename=pandas_profiling-2.5.0-py2.py3-none-any.whl size=241328 sha256=81645786541ffd480165a08412ae9302597a77a0b75080c2a306a4b9b5cf3ff0
      Stored in directory: /home/ach/.cache/pip/wheels/9b/c9/f1/4a2f30c760e017f3e2f46be999c4597a93d126ef5ea38e276f
      Building wheel for confuse (setup.py) ... [?25ldone
    [?25h  Created wheel for confuse: filename=confuse-1.0.0-cp37-none-any.whl size=17486 sha256=e026affd3264b9f5ec11bce170d12b9b2e7d35564dad4957966a484c2ee658c8
      Stored in directory: /home/ach/.cache/pip/wheels/b0/b2/96/2074eee7dbf7b7df69d004c9b6ac4e32dad04fb7666cf943bd
      Building wheel for visions (setup.py) ... [?25ldone
    [?25h  Created wheel for visions: filename=visions-0.2.2-cp37-none-any.whl size=53059 sha256=91b20c5dee9e8de062c249199d4ae8333bfc52bb0ac285b077cf2b579a4dbdc1
      Stored in directory: /home/ach/.cache/pip/wheels/53/87/68/294a9e88d82e395b38571df18f7cb71e9ab51cedae77dd6f31
      Building wheel for htmlmin (setup.py) ... [?25ldone
    [?25h  Created wheel for htmlmin: filename=htmlmin-0.1.12-cp37-none-any.whl size=27086 sha256=4f6462cc3df69d498e9cc4e82a6b6f95ce4260c877fa399c95fe5e58aac538dc
      Stored in directory: /home/ach/.cache/pip/wheels/43/07/ac/7c5a9d708d65247ac1f94066cf1db075540b85716c30255459
      Building wheel for tangled-up-in-unicode (setup.py) ... [?25ldone
    [?25h  Created wheel for tangled-up-in-unicode: filename=tangled_up_in_unicode-0.0.3-cp37-none-any.whl size=1554154 sha256=488f9b182f99de184f24acbede23c3815895eb9bf8ef82c9ce0bf54244dd88f3
      Stored in directory: /home/ach/.cache/pip/wheels/c4/57/cc/5f58206efb00418d4dcae8d08a3cb40627778ea29622f664c6
      Building wheel for kaggle (setup.py) ... [?25ldone
    [?25h  Created wheel for kaggle: filename=kaggle-1.5.6-cp37-none-any.whl size=72859 sha256=76f15a9764ed594edac2aece2fe5343cc97846e1b52f320ecac22264aaeecb90
      Stored in directory: /home/ach/.cache/pip/wheels/57/4e/e8/bb28d035162fb8f17f8ca5d42c3230e284c6aa565b42b72674
      Building wheel for attr (setup.py) ... [?25ldone
    [?25h  Created wheel for attr: filename=attr-0.3.1-cp37-none-any.whl size=2459 sha256=aacfe672fb79a445977dc333fc8c8e87c0c512178596bc7c1c7edb7e1a3f8287
      Stored in directory: /home/ach/.cache/pip/wheels/f0/96/9b/1f8892a707d17095b5a6eab0275da9d39e68e03a26aee2e726
      Building wheel for python-slugify (setup.py) ... [?25ldone
    [?25h  Created wheel for python-slugify: filename=python_slugify-4.0.0-py2.py3-none-any.whl size=5487 sha256=f2b8d0ae5d60817e937bf9a32908a299373aefcd2f80e18492dd929c1cd854fd
      Stored in directory: /home/ach/.cache/pip/wheels/11/94/81/312969455540cb0e6a773e5d68a73c14128bfdfd4a7969bb4f
    Successfully built pandas-profiling confuse visions htmlmin tangled-up-in-unicode kaggle attr python-slugify
    [31mERROR: hyperopt 0.2.3 has requirement networkx==2.2, but you'll have networkx 2.4 which is incompatible.[0m
    Installing collected packages: scipy, pandas, confuse, jinja2, tangled-up-in-unicode, attr, visions, htmlmin, pytest-pylint, joblib, phik, astropy, tqdm, text-unidecode, python-slugify, kaggle, pandas-profiling, typed-ast
      Found existing installation: scipy 1.3.1
        Uninstalling scipy-1.3.1:
          Successfully uninstalled scipy-1.3.1
      Found existing installation: pandas 0.25.1
        Uninstalling pandas-0.25.1:
          Successfully uninstalled pandas-0.25.1
      Found existing installation: Jinja2 2.10.3
        Uninstalling Jinja2-2.10.3:
          Successfully uninstalled Jinja2-2.10.3
      Found existing installation: joblib 0.13.2
        Uninstalling joblib-0.13.2:
          Successfully uninstalled joblib-0.13.2
      Found existing installation: astropy 3.2.2
        Uninstalling astropy-3.2.2:
          Successfully uninstalled astropy-3.2.2
      Found existing installation: tqdm 4.36.1
        Uninstalling tqdm-4.36.1:
          Successfully uninstalled tqdm-4.36.1
    Successfully installed astropy-4.0 attr-0.3.1 confuse-1.0.0 htmlmin-0.1.12 jinja2-2.11.1 joblib-0.14.1 kaggle-1.5.6 pandas-0.25.3 pandas-profiling-2.5.0 phik-0.9.9 pytest-pylint-0.15.1 python-slugify-4.0.0 scipy-1.4.1 tangled-up-in-unicode-0.0.3 text-unidecode-1.3 tqdm-4.42.0 typed-ast-1.4.1 visions-0.2.2

pfr = pandas_profiling.ProfileReport(Train_data)
    pfr.to_file("./example.html")

HBox(children=(FloatProgress(value=0.0, description='variables', max=29.0, style=ProgressStyle(descrip

所给出的EDA步骤为广为普遍的步骤，在实际的不管是工程还是比赛过程中，这只是最开始的一步，也是最基本的一步。
    
    接下来一般要结合模型的效果以及特征工程等来分析数据的实际建模情况，根据自己的一些理解，查阅文献，对实际问题做出判断和深入的理解。
    
    最后不断进行EDA与数据处理和挖掘，来到达更好的数据结构和分布以及较为强势相关的特征
    
    ---
    数据探索在机器学习中我们一般称为EDA（Exploratory Data Analysis）：
    
    > 是指对已有的数据（特别是调查或观察得来的原始数据）在尽量少的先验假定下进行探索，通过作图、制表、方程拟合、计算特征量等手段探索数据的结构和规律的一种数据分析方法。
    
    数据探索有利于我们发现数据的一些特性，数据之间的关联性，对于后续的特征构建是很有帮助的。
    
    1. 对于数据的初步分析（直接查看数据，或.sum(), .mean()，.descirbe()等统计函数）可以从：样本数量，训练集数量，是否有时间特征，是否是时许问题，特征所表示的含义（非匿名特征），特征类型（字符类似，int，float，time），特征的缺失情况（注意缺失的在数据中的表现形式，有些是空的有些是”NAN”符号等），特征的均值方差情况。
    
    2. 分析记录某些特征值缺失占比30%以上样本的缺失处理，有助于后续的模型验证和调节，分析特征应该是填充（填充方式是什么，均值填充，0填充，众数填充等），还是舍去，还是先做样本分类用不同的特征模型去预测。
    
    3. 对于异常值做专门的分析，分析特征异常的label是否为异常值（或者偏离均值较远或者事特殊符号）,异常值是否应该剔除，还是用正常值填充，是记录异常，还是机器本身异常等。
    
    4. 对于Label做专门的分析，分析标签的分布情况等。
    
    5. 进步分析可以通过对特征作图，特征和label联合做图（统计图，离散图），直观了解特征的分布情况，通过这一步也可以发现数据之中的一些异常值等，通过箱型图分析一些特征值的偏离情况，对于特征和特征联合作图，对于特征和label联合作图，分析其中的一些关联性。