GitHub--logparser(日志解析器)

以你之姓@ 2022-11-05 03:13 10阅读 0赞

## Apache HTTPD和NGINX访问日志解析器 ##

这是一个Logparsing框架，旨在简化[Apache HTTPD][]和[NGINX][]访问日志文件的解析。  
基本思想是，您应该能够拥有一个解析器，可以通过简单地告诉该行写入了哪些配置选项来构造该解析器。这些配置选项是访问日志行的架构。  
github地址：[https://github.com/nielsbasjes/logparser][https_github.com_nielsbasjes_logparser]  
**idea安装插件**  
Lombok能以简单的注解形式来简化java代码，提高开发人员的开发效率。例如开发中经常需要写的javabean，都需要花时间去添加相应的getter/setter，也许还要去写构造器、equals等方法，而且需要维护，当属性多时会出现大量的getter/setter方法，这些显得很冗长也没有太多技术含量，一旦修改属性，就容易出现忘记修改对应方法的失误。

Lombok能通过注解的方式，在编译时自动为属性生成构造器、getter/setter、equals、hashcode、toString方法。出现的神奇就是在源码中没有getter和setter方法，但是在编译生成的字节码文件中有getter和setter方法。这样就省去了手动重建这些代码的麻烦，使代码看起来更简洁些。  
**安装步骤：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70]  
**导入依赖**

<dependency>
        <groupId>nl.basjes.parse.httpdlog</groupId>
        <artifactId>httpdlog-parser</artifactId>
        <version>5.2</version>
    </dependency>

**nginx日志样本**  
在nginx的conf目录下的nginx.conf文件中可以配置日志打印的格式，如下：

#log_format  main   '$remote_addr - $remote_user [$time_local] [$msec] 	
     						[$request_time] [$http_host] "$request" '
                        '$status $body_bytes_sent "$request_body" "$http_referer" '
                        '"$http_user_agent" $http_x_forwarded_for'

$remote\_addr 对应客户端的地址  
$remote\_user 是请求客户端请求认证的用户名，如果没有开启认证模块的话是值为空。  
$time\_local 表示nginx服务器时间  
$msec 访问时间与时区字符串形式  
$request\_time 请求开始到返回时间  
$http\_host 请求域名  
$request 请求的url与http协议  
$status 请求状态,如成功200  
$body\_bytes\_sent 表示从服务端返回给客户端的body数据大小  
$request\_body 访问url时参数  
$http\_referer 记录从那个页面链接访问过来的  
$http\_user\_agent 记录客户浏览器的相关信息  
$http\_x\_forwarded\_for 请求转发f过来的地址  
$upstream\_response\_time:从 Nginx 建立连接 到 接收完数据并关闭连接  
**范例:**

125.88.xxx.xx - - [02/Nov/2018:14:28:49 +0800] [1541140129.431] [0.095] [ma.xx.game.com] 
    "POST /?log3/gameReport HTTP/1.1" 200 358 "type=1&id=NaN" "http://ma.xx.game.com/?log3/gameReport2" "Mozilla/5.0 (Windows NT 6.1;
     Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" -

**点击日流日志样本**

2001-980:91c0:1:8d31:a232:25e5:85d 222.68.172.190 - [05/Sep/2010:11:27:50 +0200] \"GET /images/my.jpg HTTP/1.1\" 404 23617
     \"http://www.angularjs.cn/A00n\" \"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; nl-nl) AppleWebKit/533.17.8 (KHTML, like Gecko) Version/5.0.1
      Safari/533.17.8\" \"jquery-ui-theme=Eggplant; BuI=SomeThing; Apache=127.0.0.1.1351111543699529\" \"beijingshi\"

**定义格式化字符串**

%u %h %l %t \"%r\" %>s %b \"%{ Referer}i\" \"%{ User-Agent}i\" \"%{ Cookie}i\" \"%{ Addr}i\"

**使用方式**  
打印日志格式字符串的所有参数  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 2]

import nl.basjes.parse.core.Field;
    
    import java.util.HashMap;
    import java.util.Map;
    import java.util.TreeSet;
    
    public class MyRecord { 
    
        private final Map<String, String> results = new HashMap<>(32);
    
        @Field("STRING:request.firstline.uri.query.*")
        public void setQueryDeepMany(final String name, final String value) { 
            results.put(name, value);
        }
    
        @Field("STRING:request.firstline.uri.query.img")
        public void setQueryImg(final String name, final String value) { 
            results.put(name, value);
        }
    
        @Field("IP:connection.client.host")
        public void setIP(final String value) { 
            results.put("IP:connection.client.host", value);
        }
    
        @Field({ 
                "HTTP.QUERYSTRING:request.firstline.uri.query",
                "NUMBER:connection.client.logname",
                "STRING:connection.client.user",
                "TIME.STAMP:request.receive.time",
                "HTTP.URI:request.firstline.uri",
                "BYTESCLF:response.body.bytes",
                "HTTP.URI:request.referer",
                "HTTP.USERAGENT:request.user-agent",
                "TIME.DAY:request.receive.time.day",
                "TIME.HOUR:request.receive.time.hour",
                "TIME.MONTHNAME:request.receive.time.monthname"
        })
        public void setValue(final String name, final String value) { 
            results.put(name, value);
        }
    
        public String toString() { 
            StringBuilder sb = new StringBuilder();
            TreeSet<String> keys = new TreeSet<>(results.keySet());
            for (String key : keys) { 
                sb.append(key).append(" = ").append(results.get(key)).append('\n');
            }
    
            return sb.toString();
        }
    
        public void clear() { 
            results.clear();
        }
    }

import nl.basjes.parse.core.Parser;
    import nl.basjes.parse.core.exceptions.DissectionFailure;
    import nl.basjes.parse.core.exceptions.InvalidDissectorException;
    import nl.basjes.parse.core.exceptions.MissingDissectorsException;
    import nl.basjes.parse.httpdlog.HttpdLoglineParser;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    import java.util.List;
    
    public final class Main { 
        private Main(){ }
    
        private static final Logger LOG = LoggerFactory.getLogger(Main.class);
    
        /** * 打印出来解析后的所有参数 * @param logformat * @throws NoSuchMethodException * @throws MissingDissectorsException * @throws InvalidDissectorException */
        private void printAllPossibles(String logformat) throws NoSuchMethodException, MissingDissectorsException, InvalidDissectorException { 
            // To figure out what values we CAN get from this line we instantiate the parser with a dummy class
            // that does not have ANY @Field annotations.
            Parser<Object> dummyParser= new HttpdLoglineParser<>(Object.class, logformat);
    
            List<String> possiblePaths;
            possiblePaths = dummyParser.getPossiblePaths();
    
            // If you want to call 'getCasts' then the actual parser needs to be constructed.
            // Simply calling getPossiblePaths does not build the actual parser.
            // Because we want this for all possibilities yet we are never actually going to use this instance of the parser
            // We simply give it a random method with the right signature and tell it we want all possible paths
            dummyParser.addParseTarget(String.class.getMethod("indexOf", String.class), possiblePaths);
    
            System.out.println("==================================");
            System.out.println("Possible output:");
            for (String path : possiblePaths) { 
                System.out.println(path+" "+dummyParser.getCasts(path));
            }
            LOG.info("==================================");
        }
    
        private void run() throws InvalidDissectorException, MissingDissectorsException, NoSuchMethodException, DissectionFailure { 
    
            // This format and logline originate from here:
            // https://stackoverflow.com/questions/20349184/java-parse-log-file
            String logformat = "%t %u [%D %h %{True-Client-IP}i %{UNIQUE_ID}e %r] %{Cookie}i %s \"%{User-Agent}i\" \"%{host}i\" %l %b %{Referer}i";
            String logline = "[02/Dec/2013:14:10:30 -0000] - [52075 10.102.4.254 177.43.52.210 UpyU1gpmBAwAACfd5W0AAAAW GET /SS14-VTam-ny_019.j" +
                    "pg.rendition.zoomable.jpg HTTP/1.1] hsfirstvisit=http%3A%2F%2Fwww.domain.com%2Fen-us||1372268254000; _opt_vi_3FNG8DZU=F870" +
                    "DCFD-CBA4-4B6E-BB58-4605A78EE71A; __ptca=145721067.0aDxsZlIuM48.1372279055.1379945057.1379950362.9; __ptv_62vY4e=0aDxsZlIu" +
                    "M48; __pti_62vY4e=0aDxsZlIuM48; __ptcz=145721067.1372279055.1.0.ptmcsr=(direct)|ptmcmd=(none)|ptmccn=(direct); __hstc=1457" +
                    "21067.b86362bb7a1d257bfa2d1fb77e128a85.1372268254968.1379934256743.1379939561848.9; hubspotutk=b86362bb7a1d257bfa2d1fb77e1" +
                    "28a85; USER_GROUP=julinho%3Afalse; has_js=1; WT_FPC=id=177.43.52.210-1491335248.30301337:lv=1385997780893:ss=1385997780893" +
                    "; dtCookie=1F2E0E1037589799D8D503EB8CFA12A1|_default|1; RM=julinho%3A5248423ad3fe062f06c54915e6cde5cb45147977; wcid=UpyKsQ" +
                    "pmBAwAABURyNoAAAAS%3A35d8227ba1e8a9a9cebaaf8d019a74777c32b4c8; Carte::KerberosLexicon_getWGSN=82ae3dcd1b956288c3c86bdbed6e" +
                    "bcc0fd040e1e; UserData=Username%3AJULINHO%3AHomepage%3A1%3AReReg%3A0%3ATrialist%3A0%3ALanguage%3Aen%3ACcode%3Abr%3AForceRe" +
                    "Reg%3A0; UserID=1356673%3A12345%3A1234567890%3A123%3Accode%3Abr; USER_DATA=1356673%3Ajulinho%3AJulio+Jose%3Ada+Silva%3Ajul" +
                    "inho%40tecnoblu.com.br%3A0%3A1%3Aen%3Abr%3A%3AWGSN%3A1385990833.81925%3A82ae3dcd1b956288c3c86bdbed6ebcc0fd040e1e; MODE=FON" +
                    "TIS; SECTION=%2Fcontent%2Fsection%2Fhome.html; edge_auth=ip%3D177.43.52.210~expires%3D1385994522~access%3D%2Fapps%2F%2A%21" +
                    "%2Fbin%2F%2A%21%2Fcontent%2F%2A%21%2Fetc%2F%2A%21%2Fhome%2F%2A%21%2Flibs%2F%2A%21%2Freport%2F%2A%21%2Fsection%2F%2A%21%2Fw" +
                    "gsn%2F%2A~md5%3D90e73ee10161c1afacab12c6ea30b4ef; __utma=94539802.1793276213.1372268248.1385572390.1385990581.16; __utmb=9" +
                    "4539802.52.9.1385991739764; __utmc=94539802; __utmz=94539802.1372268248.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);" +
                    " WT_FPC=id=177.43.52.210-1491335248.30301337:lv=1386000374581:ss=1386000374581; dtPC=-; NSC_wtfswfs_xfcgbsn40-41=ffffffff0" +
                    "96e1a1d45525d5f4f58455e445a4a423660; akamai-edge=5ac6e5b3d0bbe2ea771bb2916d8bab34ea222a6a 200 \"Mozilla/5.0 (Windows NT 6." +
                    "2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36\" \"www.domain.com\" - 463952 http://ww" +
                    "w.domain.com/content/report/shows/New_York/KSHK/trip/s_s_14_ny_ww/sheers.html";
    
            //打印出来所有参数
            printAllPossibles(logformat);
    
            //定义日志的解析器，传递javabean对象和日志解析规则
            Parser<MyRecord> parser = new HttpdLoglineParser<>(MyRecord.class, logformat);
            //实例化javabean
            MyRecord record = new MyRecord();
    
            System.out.println("==================================================================================");
            //解析数据，将解析数据后的数据给javabean
            parser.parse(record, logline);
            System.out.println(record.toString());
            System.out.println("==================================================================================");
        }
    
        public static void main(final String[] args) throws Exception { 
            new Main().run();
        }
    
    }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 3]

[Apache HTTPD]: https://httpd.apache.org/
[NGINX]: https://nginx.org/
[https_github.com_nielsbasjes_logparser]: https://github.com/nielsbasjes/logparser
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70]: /images/20221023/928a635e0cef4b51bc9b424c6e5a1982.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 1]: /images/20221023/37dcad69b19b453392c4ec8f0924352b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 2]: /images/20221023/1d09863df7b64a8eae789a9f9585ade0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 3]: /images/20221023/d95b2565a425400c9de3fb531e8f0562.png