centos6.9安装systemtap+ FlameGraph 对nginx内存占用生成火焰图

缺乏、安全感 2021-09-07 06:05 428阅读 0赞

一台centos上安装了nginx 1.8.0，现需要对其内存使用情况进行查看，具体操作步骤如下。

查看centos的内核版本：

# uname -a
    Linux online123.virtual 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

### 1、下载systemTap依赖： ###

下载kernel-devel-\[版本号\]、kernel-debuginfo-\[版本号\]、kernel-debuginfo-common-\[版本号\]，这里的版本号要和内核的一致，例如：2.6.32-642.el6.x86\_64

如何找到下载地址？可以直接在google上输入kernel-debuginfo-common-2.6.32-642.el6.x86\_64.rpm，可以找到很多源，例如：

[https://oss.oracle.com/el6/debuginfo/][https_oss.oracle.com_el6_debuginfo]

[http://mirrors.ocf.berkeley.edu/centos-debuginfo/6/x86\_64/][http_mirrors.ocf.berkeley.edu_centos-debuginfo_6_x86_64]

下载后，依次装kernel-debuginfo-common、kernel-debuginfo、kernel-devel

# rpm -ivh kernel-debuginfo-common-x86_64-2.6.32-642.el6.x86_64.rpm 
    warning: kernel-debuginfo-common-x86_64-2.6.32-642.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
    Preparing...                ########################################### [100%]
       1:kernel-debuginfo-common########################################### [100%]
    
    # rpm -ivh kernel-debuginfo-2.6.32-642.el6.x86_64.rpm 
    warning: kernel-debuginfo-2.6.32-642.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
    Preparing...                ########################################### [100%]
       1:kernel-debuginfo       #########################################   ( 94%)
    ########################################### [100%]
    
    # rpm -ivh kernel-devel-2.6.32-642.el6.x86_64.rpm
    warning: kernel-devel-2.6.32-642.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 1d1e034b: NOKEY
    Preparing...                ########################################### [100%]
       1:kernel-devel           ########################################### [100%]

1）如果kernel-devel之前安装的版本不对，需要先卸载，否则会报错，使用rpm -e --nodeps

# rpm -ivh kernel-devel-2.6.32-642.el6.x86_64.rpm
    warning: kernel-devel-2.6.32-642.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 1d1e034b: NOKEY
    Preparing...                ########################################### [100%]
    	package kernel-devel-2.6.32-642.15.1.el6.x86_64 (which is newer than kernel-devel-2.6.32-642.el6.x86_64) is already installed
        
    #卸载
    # rpm -e kernel-devel-2.6.32-642.15.1.el6.x86_64
    error: Failed dependencies:
    	kernel-devel is needed by (installed) systemtap-devel-2.9-4.el6.x86_64
    # rpm -e --nodeps kernel-devel-2.6.32-642.15.1.el6.x86_64

2）必须先安装kernel-debuginfo-common，再装kernel-debuginfo，否则会报如下错误：

# rpm -ivh kernel-debuginfo-2.6.32-642.el6.x86_64.rpm 
    warning: kernel-debuginfo-2.6.32-642.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
    error: Failed dependencies:
    	kernel-debuginfo-common-x86_64 = 2.6.32-642.el6 is needed by kernel-debuginfo-2.6.32-642.el6.x86_64

3）如果只安装了kernel-devle，没有安装kernel-debuginfo和kernel-debuginfo-common，直接安装systemTap，然后用stapxx-master工具，那么在使用时会提示没有“矮人”（dwarf）错误：

# export PATH=$PWD:$PATH
    # ./samples/sample-bt-leaks.sxx -x 4739 --arg time=5 -D STP_NO_OVERLOAD -D MAXMAPENTRIES=10000 > a.bt
    Found exact match for libc: /lib64/libc-2.12.so
    WARNING: cannot find module /lib64/libc-2.12.so debuginfo: No DWARF information found [man warning::debuginfo]
    semantic error: while processing probe process("/lib64/libc-2.12.so").function("malloc").return from: process("/lib64/libc-2.12.so").function("malloc").return
    
    semantic error: type definition 'size_t' not found in '/lib64/libc-2.12.so': operator '@cast' at <input>:58:28
            source:                 p = ptr - &@cast(0, "size_t")[1]
                                               ^
    
    Pass 2: analysis failed.  [man error::pass2]
    Number of similar warning messages suppressed: 35.
    Rerun with -v to see them.

### 2、安装systemTap： ###

`SystemTap` 通过将脚本语句翻译成`C`语句，编译成内核模块。模块加载之后,依据时间处理机制来收集有关正在运行的`Linux`系统的信息。安装如下；

yum -y install systemtap systemtap-runtime

注意：可以通过yum remove systemtap 卸载。

安装之后可以通过如下语句验证：

stap -ve 'probe begin { log("hello world") exit() }'

###  3、stapxx-master采集信息： ###

要知道`systemtap`只是个内核探测工具，需要我们自己写脚本埋点。stap++ 是一个简单的微语言，用来扩展systemtap（Simple macro language extensions to systemtap），github地址：[https://github.com/openresty/stapxx][https_github.com_openresty_stapxx]

春哥已经写好了一些常用的脚本工具：

*  [https://github.com/agentzh/stapxx\#ngx-lj-gc][https_github.com_agentzh_stapxx_ngx-lj-gc] ：Lua VM 分配的空间
 *  [https://github.com/agentzh/stapxx\#ngx-lj-gc-objs][https_github.com_agentzh_stapxx_ngx-lj-gc-objs] ：Lua VM 分配的空间
 *  [https://github.com/agentzh/stapxx\#sample-bt-leaks][https_github.com_agentzh_stapxx_sample-bt-leaks] ：内存泄漏火焰图

注意：如果使用的是openresty，可以直接使用openresty-systemtap-toolkit这个中的工具，地址：[https://github.com/openresty/openresty-systemtap-toolkit][https_github.com_openresty_openresty-systemtap-toolkit]

1）下载：

# wget https://github.com/openresty/stapxx/archive/master.zip
    # unzip master.zip

2）采集内存信息：

# cd stapxx-master
    # export PATH=$PWD:$PATH
    # ./samples/sample-bt-leaks.sxx -x 4739 --arg time=5 -D STP_NO_OVERLOAD -D MAXMAPENTRIES=10000 > a.bt

注意：如果没有安装kernel-devel会报如下错误

# cd stapxx-master
    # export PATH=$PWD:$PATH
    # ./samples/sample-bt-leaks.sxx -x 4739 --arg time=5 -D STP_NO_OVERLOAD -D MAXMAPENTRIES=10000 > a.bt
    Found exact match for libc: /lib64/libc-2.12.so
    Checking "/lib/modules/2.6.32-642.el6.x86_64/build/.config" failed with error: No such file or directory
    Checking "/lib/modules/2.6.32-642.el6.x86_64/build/.config" failed with error: No such file or directory
    Incorrect version or missing kernel-devel package, use: yum install kernel-devel-2.6.32-642.el6.x86_64

3）可能会报如下错误：

解决方法：首先查看glibc：

# rpm -aq | grep glibc
    glibc-common-2.12-1.192.el6.x86_64
    glibc-headers-2.12-1.192.el6.x86_64
    glibc-devel-2.12-1.192.el6.x86_64
    glibc-2.12-1.192.el6.x86_64

然后下载对应的glibc-debuginfo-common-\[版本\]、glibc-debuginfo-\[版本\]，其中版本要和glibc的一致，例如：2.12-1.192.el6.x86\_64

如何下载glibc-debuginfo-common-2.12-1.192.el6.x86\_64？方法也是直接在google输入glibc-debuginfo-common-2.12-1.192.el6.x86\_64，会得到很多源，找一个下载即可。例如：[http://debuginfo.centos.org/6/x86\_64/][http_debuginfo.centos.org_6_x86_64]

下载后，依次安装glibc-debuginfo-common、glibc-debuginfo：

# rpm -iv glibc-debuginfo-common-2.12-1.192.el6.x86_64.rpm 
    warning: glibc-debuginfo-common-2.12-1.192.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
    Preparing packages for installation...
    glibc-debuginfo-common-2.12-1.192.el6
    
    # rpm -ivh glibc-debuginfo-2.12-1.192.el6.x86_64.rpm 
    warning: glibc-debuginfo-2.12-1.192.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
    Preparing...                ########################################### [100%]
       1:glibc-debuginfo        ########################################### [100%]

### 4、生成火焰图： ###

1）下载FlameGraph：

地址：[https://github.com/brendangregg/FlameGraph][https_github.com_brendangregg_FlameGraph]

下载后直接unzip解压即可。

2）生成火焰图：

# cd stapxx-master
    # export PATH=$PWD:$PATH
    # ./samples/sample-bt-leaks.sxx -x 16795 --arg time=5 -D STP_NO_OVERLOAD -D MAXMAPENTRIES=10000 > a.bt
    WARNING: Start tracing 16795 (/opt/nginx/sbin/nginx)...
    WARNING: Wait for 5 sec to complete.
    
    # export PATH=/path/to/FlameGraph:$PATH
    # stackcollapse-stap.pl a.bt > a.cbt
    # flamegraph.pl --countname=bytes \
            --title="Memory Leak Flame Graph" a.cbt > a.svg

3）其他：

上面使用了systemtap+flamegraph生成了火焰图，也可以使用perf+FlameGraph生成火焰图。

### 5、openresty-systemtap-toolkit采集信息： ###

同样，systemtap只是个内核探测工具，不只是可以用在openresty中的，你得自己写好脚本来进行埋点等工作。但是春哥已经在 openresty-systemtap-toolkit 中提供了许多可以直接使用的脚本，我们直接拿过来用就好了，毕竟我自己是不会写的。openresty-systemtap-toolkit支持：

*  [https://github.com/agentzh/nginx-systemtap-toolkit\#ngx-sample-lua-bt][https_github.com_agentzh_nginx-systemtap-toolkit_ngx-sample-lua-bt]：生成lua级别的火焰图
 *  [https://github.com/agentzh/nginx-systemtap-toolkit\#ngx-cycle-pool][https_github.com_agentzh_nginx-systemtap-toolkit_ngx-cycle-pool] ： 分析最大的nginx进程
 *  [https://github.com/agentzh/nginx-systemtap-toolkit\#ngx-active-reqs][https_github.com_agentzh_nginx-systemtap-toolkit_ngx-active-reqs] ： nginx worker 进程中每请求的内存池的内存使用情况

参考：

[https://groups.google.com/g/openresty/c/2LxTiBoJJkk?pli=1][https_groups.google.com_g_openresty_c_2LxTiBoJJkk_pli_1]

[https://www.cnblogs.com/pk-tiger/p/11757716.html][https_www.cnblogs.com_pk-tiger_p_11757716.html]

[https://segmentfault.com/a/1190000009592315][https_segmentfault.com_a_1190000009592315]

[http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html][http_www.brendangregg.com_FlameGraphs_memoryflamegraphs.html]

[https://blog.csdn.net/tao\_627/article/details/52137901][https_blog.csdn.net_tao_627_article_details_52137901]

[https_oss.oracle.com_el6_debuginfo]: https://oss.oracle.com/el6/debuginfo/
[http_mirrors.ocf.berkeley.edu_centos-debuginfo_6_x86_64]: http://mirrors.ocf.berkeley.edu/centos-debuginfo/6/x86_64/
[https_github.com_openresty_stapxx]: https://github.com/openresty/stapxx
[https_github.com_agentzh_stapxx_ngx-lj-gc]: https://github.com/agentzh/stapxx#ngx-lj-gc
[https_github.com_agentzh_stapxx_ngx-lj-gc-objs]: https://github.com/agentzh/stapxx#ngx-lj-gc-objs
[https_github.com_agentzh_stapxx_sample-bt-leaks]: https://github.com/agentzh/stapxx#sample-bt-leaks
[https_github.com_openresty_openresty-systemtap-toolkit]: https://github.com/openresty/openresty-systemtap-toolkit
[http_debuginfo.centos.org_6_x86_64]: http://debuginfo.centos.org/6/x86_64/
[https_github.com_brendangregg_FlameGraph]: https://github.com/brendangregg/FlameGraph
[https_github.com_agentzh_nginx-systemtap-toolkit_ngx-sample-lua-bt]: https://github.com/agentzh/nginx-systemtap-toolkit#ngx-sample-lua-bt
[https_github.com_agentzh_nginx-systemtap-toolkit_ngx-cycle-pool]: https://github.com/agentzh/nginx-systemtap-toolkit#ngx-cycle-pool
[https_github.com_agentzh_nginx-systemtap-toolkit_ngx-active-reqs]: https://github.com/agentzh/nginx-systemtap-toolkit#ngx-active-reqs
[https_groups.google.com_g_openresty_c_2LxTiBoJJkk_pli_1]: https://groups.google.com/g/openresty/c/2LxTiBoJJkk?pli=1
[https_www.cnblogs.com_pk-tiger_p_11757716.html]: https://www.cnblogs.com/pk-tiger/p/11757716.html
[https_segmentfault.com_a_1190000009592315]: https://segmentfault.com/a/1190000009592315
[http_www.brendangregg.com_FlameGraphs_memoryflamegraphs.html]: http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html
[https_blog.csdn.net_tao_627_article_details_52137901]: https://blog.csdn.net/tao_627/article/details/52137901