Python+pandas对系统错误进行日志分析

本是古典何须时尚 2023-02-17 14:51 21阅读 0赞

## Python+pandas对系统错误进行日志分析 ##

生产日志一般都很大，即使就是一天有时也是很大的，如果想要对错误日志进行统计，在bash shell里处理相当费时，使用pandas处理，千万级别的日志，也不会超过1分钟。  
我的生产日志是这样的：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70]  
现在要分别统计错误代码是0-9的错误次数，代码这样实现：

#coding=gbk
    import pandas as pd
    log=pd.read_table('my.log', sep='\n', iterator=True,encoding='gbk',header=None,names=['desc'])
    loop = True
    chunkSize = 10000
    chunks = []
    while loop:
        try:
            chunk = log.get_chunk(chunkSize)
            chunks.append(chunk)
        except StopIteration:
            loop=False
            
    log=pd.concat(chunks,ignore_index=False)
    print('错误 0='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 0',:].count()['desc'])))
    print('错误 1='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 1',:].count()['desc'])))
    print('错误 2='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 2',:].count()['desc'])))
    print('错误 3='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 3',:].count()['desc'])))
    print('错误 4='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 4',:].count()['desc'])))
    print('错误 5='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 5',:].count()['desc'])))
    print('错误 6='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 6',:].count()['desc'])))
    print('错误 7='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 7',:].count()['desc'])))
    print('错误 8='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 8',:].count()['desc'])))
    print('错误 9='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 9',:].count()['desc'])))

介绍下pd.read\_table()，感兴趣可以研究下每个参数使用：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70 2]

图解Python数据结构与算法-实战篇

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200614170702305.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200614171029636.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020061417110728.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc4MDA4MA==,size_16,color_FFFFFF,t_70