nginx前台启动-段错误调试

分手后的思念是犯贱 2022-05-17 10:40 1719阅读 0赞

nginx.conf中

daemon on|off

master\_process on|off

**在调试Nginx功能的时候，出现如下问题：**

2017/02/27 16:23:50 \[notice\] 13604\#0: signal 17 (SIGCHLD) received  
2017/02/27 16:23:50 \[alert\] 13604\#0: worker process 13605 exited on signal 11 (core dumped)  
2017/02/27 16:23:50 \[notice\] 13604\#0: start worker process 13816 0  
2017/02/27 16:23:50 \[notice\] 13604\#0: signal 29 (SIGIO) received

发现有子进程发生了core dump。

但是在nginx/logs目录下面执行

ulimit -c unlimited

还是没有core文件产生。

解决方法是在nginx.conf文件中 添加

worker\_rlimit\_core 10000m;  
working\_directory /usr/local/nginx/logs;

重新启动Nginx，复现问题。

相应的core文件出现在logs目录下

**结合lua时候可能不生产段错误：调试方法如下。**  
\[root@WiseGrid sbin\]\# gdb smartl7  
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7  
Copyright (C) 2013 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"  
and "show warranty" for details.  
This GDB was configured as "x86\_64-redhat-linux-gnu".  
For bug reporting instructions, please see:  
<http://www.gnu.org/software/gdb/bugs/>...  
Reading symbols from /opt/WiseGrid/smartl7/sbin/smartl7\_normal...(no debugging symbols found)...done.  
(gdb) run  
Starting program: /opt/WiseGrid/smartl7/sbin/smartl7   
\[Thread debugging using libthread\_db enabled\]  
Using host libthread\_db library "/lib64/libthread\_db.so.1".

Program received signal SIGSEGV, Segmentation fault.  
0x0000000000543aeb in ngx\_http\_lua\_ngx\_error\_page ()  
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86\_64 lua-5.1.4-15.el7.x86\_64 nss-softokn-freebl-3.28.3-8.el7\_4.x86\_64 pcre-8.32-17.el7.x86\_64 zlib-1.2.7-17.el7.x86\_64

**\[root@WiseGrid sbin\]\# yum install yum-utils**  
**安装debuginfo-install glibc-2.17-196.el7.x86\_64 lua-5.1.4-15.el7.x86\_64 nss-softokn-freebl-3.28.3-8.el7\_4.x86\_64 pcre-8.32-17.el7.x86\_64 zlib-1.2.7-17.el7.x86\_64**  
Program received signal SIGSEGV, Segmentation fault.  
0x0000000000543aeb in ngx\_http\_lua\_ngx\_error\_page ()  
(gdb) bt  
\#0  0x0000000000543aeb in ngx\_http\_lua\_ngx\_error\_page ()  
\#1  0x00007ffff7766324 in luaD\_precall (L=L@entry=0xbe5500, func=func@entry=0xc03c70, nresults=nresults@entry=0) at ldo.c:319  
\#2  0x00007ffff7770e57 in luaV\_execute (L=L@entry=0xbe5500, nexeccalls=nexeccalls@entry=1) at lvm.c:590  
\#3  0x00007ffff776674d in luaD\_call (L=0xbe5500, func=0xc03c60, nResults=<optimized out>) at ldo.c:377  
\#4  0x00007ffff7765a6e in luaD\_rawrunprotected (L=L@entry=0xbe5500, f=f@entry=0x7ffff7761050 <f\_call>, ud=ud@entry=0x7fffffffdb80) at ldo.c:116  
\#5  0x00007ffff77668da in luaD\_pcall (L=L@entry=0xbe5500, func=func@entry=0x7ffff7761050 <f\_call>, u=u@entry=0x7fffffffdb80, old\_top=32, ef=<optimized out>) at ldo.c:463  
\#6  0x00007ffff776244d in lua\_pcall (L=0xbe5500, nargs=0, nresults=1, errfunc=<optimized out>) at lapi.c:821  
\#7  0x0000000000567324 in ngx\_http\_lua\_header\_filter\_by\_chunk ()  
\#8  0x000000000056751f in ngx\_http\_lua\_header\_filter\_inline ()  
\#9  0x000000000056779b in ngx\_http\_lua\_header\_filter ()  
\#10 0x0000000000586648 in ngx\_http\_subs\_header\_filter ()  
\#11 0x00000000004dd510 in ngx\_http\_not\_modified\_header\_filter ()  
\#12 0x000000000049c587 in ngx\_http\_send\_header ()  
\#13 0x00000000004a5dcf in ngx\_http\_send\_special\_response ()  
\#14 0x00000000004a4ecb in ngx\_http\_special\_response\_handler ()  
\#15 0x00000000004ab450 in ngx\_http\_finalize\_request ()  
\#16 0x0000000000499eb6 in ngx\_http\_core\_rewrite\_phase ()  
\#17 0x0000000000499cc4 in ngx\_http\_core\_run\_phases ()  
\#18 0x0000000000499c32 in ngx\_http\_handler ()  
\#19 0x00000000004aa7ac in ngx\_http\_process\_request ()  
\#20 0x00000000004a90bd in ngx\_http\_process\_request\_headers ()  
\#21 0x00000000004a8497 in ngx\_http\_process\_request\_line ()  
\#22 0x00000000004a707d in ngx\_http\_wait\_request\_handler ()  
\#23 0x00000000004898d3 in ngx\_epoll\_process\_events ()  
\#24 0x0000000000479d24 in ngx\_process\_events\_and\_timers ()  
\#25 0x0000000000485ddc in ngx\_single\_process\_cycle ()  
\#26 0x000000000044eb28 in main ()

# nginx 出现 segfault at 0 ip 000000000043f750 #

2015-08-17 表现为nginx有error.log中出现：

> 2015/08/17 19:30:54 \[alert\] 21773\#0: worker process 21780 exited on signal 11

系统/var/log/messages中出现：

> Aug 17 19:24:09 BJ-ZW-9-123 kernel: nginx\[34285\]: segfault at 0 ip 000000000043f750 sp 00007fff4f309970 error 4 in nginx\[400000+299000\]

面客户端也会报错，具体是不同的客户端提示出来不一样。大致是ssl connect fail。

# 查找问题 #

从log上看，是c语言那种段错误。 一般都是使用了空指针变量之类。

## 1. 下载nginx源码进行编译 CFLAGS=”-g -O0” ./configure –with-debug ##

官方的说明在此： [http://wiki.nginx.org/Debugging][http_wiki.nginx.org_Debugging]

WTB,官方里使用worker\_rlimit\_core 500M; // 500M，实际中还不够，我的core文件要达到540多M。

## 2. 从系统打开coredump文件 ##

按照官网的方式足够打开dump core文件的，这里说一下另一种方法：

$> ulimit -c unlimited

在不限coredump文件大小后，还可以配置core文件输出目录。 不然的话，ngx默认可能输出到启动进程的目录或配置中的工作目录working\_directory；

$> mkdir /corefile     # 先建目录，还要确认nginx用户可以写此目录
    $> echo "/corefile/core-%e-%p-%h-%t" > /proc/sys/kernel/core_pattern

按以上配置后，coredump文件就会输出到/corefile 目录

1.  等待coredump文件生成与使用gdb分析：
    
        root@BJ-ZW-9-123:/corefile$ gdb /usr/local/nginx/sbin/nginx core-nginx-40162-1439808176
        GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
        Copyright (C) 2010 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-redhat-linux-gnu".
        For bug reporting instructions, please see:
        <http://www.gnu.org/software/gdb/bugs/>...
        Reading symbols from /usr/local/nginx/sbin/nginx...done.
        
        warning: exec file is newer than core file.
        [New Thread 40162]
        Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
        [Thread debugging using libthread_db enabled]
        Loaded symbols for /lib64/libpthread.so.0
        Core was generated by `nginx: worker process  '.
        Program terminated with signal 11, Segmentation fault.
        #0  0x000000000043f750 in ngx_ssl_new_session (ssl_conn=0x2464a20, sess=0x25689c0) at src/event/ngx_event_openssl.c:2309
        warning: Source file is more recent than executable.
        2309    cache = shm_zone->data;
        Missing separate debuginfos, use: debuginfo-install nginx-1.4.4-1.el6.x86_64
        (gdb)
2.  从上面最后几句中输出：
    
        src/event/ngx_event_openssl.c:2309
        warning: Source file is more recent than executable.
        2309    cache = shm_zone->data;

ngx\_event\_openssl.c文件的2309行。 很可能就是shm\_zone是NULL。打开源文件看

2306: ssl_ctx = SSL_get_SSL_CTX(ssl_conn);
    2307: shm_zone = SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_session_cache_index);
    2308:
    2309: cache = shm_zone->data;

很明显 并没有检测 SSL\_CTX\_get\_ex\_data 的返回值，其有可能是NULL。 然后直接使用了shm\_zone(可能NULL)。出现段错误，nginx崩了。 然后，对源码进行小修改后，再编译，果然在error.log出来下面的log

2308: if(shm_zone == NULL){
    2309:     ngx_log_error(NGX_LOG_ERR, c->log, 0,
    2310:          "shm_zone was NULL.%s", shpool->log_ctx);
    2311:     return 0;
    2312: }

## 4.配置的原因： ##

产生此段错误的原因是，我们整站都是使用HTTPS，对ssl的配置就下面2项：

ssl_session_cache   shared:SSL:500m;
    ssl_session_timeout 10m;

很明显，当连接数很大时，500M的共享内存不够用。然后错码没处理NULL返回. 我再搜索了一下nginx源码对此函数的使用：

root@BJ-ZW-9-123:~/nginx-1.8.0/src$ grep SSL_CTX_get_ex_data * -Rn
    event/ngx_event_openssl.c:2153:    cert = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_certificate_index);
    event/ngx_event_openssl.c:2307:    shm_zone = SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_session_cache_index);
    event/ngx_event_openssl.c:2458:    shm_zone = SSL_CTX_get_ex_data(SSL_get_SSL_CTX(ssl_conn),
    event/ngx_event_openssl.c:2551:    shm_zone = SSL_CTX_get_ex_data(ssl, ngx_ssl_session_cache_index);
    event/ngx_event_openssl.c:2845:    keys = SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_session_ticket_keys_index);
    event/ngx_event_openssl_stapling.c:198:    staple = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_stapling_index);
    event/ngx_event_openssl_stapling.c:267:    staple = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_stapling_index);
    event/ngx_event_openssl_stapling.c:268:    cert = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_certificate_index);
    event/ngx_event_openssl_stapling.c:353:    staple = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_stapling_index);
    event/ngx_event_openssl_stapling.c:440:    staple = SSL_CTX_get_ex_data(ssl->ctx, ngx_ssl_stapling_index);

## 5. 最新版nginx-1.9.3依然有此问题 ##

还真不少地方用到，自求多福吧。 以上使用的 nginx-1.8.0的代码。 但，截至2015-08-17号，nginx稳定版是nginx-1.9.3(2015-07-14发布)，并未修复此bug。此版本文件源码[http://hg.nginx.org/nginx/file/bd55d75a1410/src/event/ngx\_event\_openssl.c][http_hg.nginx.org_nginx_file_bd55d75a1410_src_event_ngx_event_openssl.c]

[http_wiki.nginx.org_Debugging]: http://wiki.nginx.org/Debugging
[http_hg.nginx.org_nginx_file_bd55d75a1410_src_event_ngx_event_openssl.c]: http://hg.nginx.org/nginx/file/bd55d75a1410/src/event/ngx_event_openssl.c