服务器宕机记录16.12.14

小咪咪 2022-07-14 14:58 793阅读 0赞

阿里云主机

16.12.14日11：37分宕机，第一时间收到短信，ssh远程远法登录，阿里云控制台，无法登录。查看CPU和内存，没有鸟用，只能重启。

11：40分，正常登录主机，查看messages

Dec 14 11:30:50 iZ258fuhh93Z systemd: Started Session 178481 of user root.  
Dec 14 11:30:50 iZ258fuhh93Z systemd: Starting Session 178481 of user root.  
Dec 14 11:30:51 iZ258fuhh93Z systemd: Started Session 178479 of user deploy.  
Dec 14 11:30:51 iZ258fuhh93Z systemd: Starting Session 178479 of user deploy.  
Dec 14 11:30:52 iZ258fuhh93Z systemd: Started Session 178480 of user deploy.  
Dec 14 11:30:57 iZ258fuhh93Z systemd: Starting Session 178480 of user deploy.  
Dec 14 11:33:41 iZ258fuhh93Z systemd-logind: **Failed to start session scope session-178482.scope: Connection timed out (null)**  
Dec 14 11:34:20 iZ258fuhh93Z systemd-logind: **Failed to abandon session scope: Connection timed out**  
Dec 14 11:35:37 iZ258fuhh93Z crond: sendmail: fatal: parameter inet\_interfaces: no local interface found for ::1  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178485 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178485 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Failed to start Session 178482 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Failed to start Session 178483 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178484 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178484 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178486 of user deploy.  
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178486 of user deploy.  
Dec 14 11:36:35 iZ258fuhh93Z systemd-logind: **Failed to abandon session scope: Connection timed out**  
Dec 14 11:36:59 iZ258fuhh93Z systemd: Started Session 178487 of user deploy.  
Dec 14 11:36:59 iZ258fuhh93Z systemd: Starting Session 178487 of user deploy.  
Dec 14 11:36:59 iZ258fuhh93Z crond: sendmail: fatal: parameter inet\_interfaces: no local interface found for ::1  
Dec 14 11:37:06 iZ258fuhh93Z crond: sendmail: fatal: parameter inet\_interfaces: no local interface found for ::1  
Dec 14 11:37:07 iZ258fuhh93Z systemd: Started Session 178488 of user deploy.  
Dec 14 11:37:07 iZ258fuhh93Z systemd: Starting Session 178488 of user deploy.  
Dec 14 11:37:13 iZ258fuhh93Z crond: sendmail: fatal: parameter inet\_interfaces: no local interface found for ::1  
Dec 14 11:37:13 iZ258fuhh93Z crond: sendmail: fatal: parameter inet\_interfaces: no local interface found for ::1  
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).  
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.  
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).  
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.  
Dec 14 11:39:34 iZ258fuhh93Z systemd: **getty@tty1.service has no holdoff time, scheduling restart.**  
Dec 14 11:39:35 iZ258fuhh93Z systemd: **Failed to fork: Cannot allocate memory**  
Dec 14 11:39:35 iZ258fuhh93Z systemd: **getty@tty2.service: Supervising process 48 which is not our child. We'll most likely not notice when it exits.**

Dec 14 11:39:39 iZ258fuhh93Z systemd: Started Session 178490 of user deploy.  
Dec 14 11:39:40 iZ258fuhh93Z systemd: Starting Session 178490 of user deploy.  
Dec 14 11:39:40 iZ258fuhh93Z systemd: Started Session 178489 of user deploy.  
Dec 14 11:39:40 iZ258fuhh93Z systemd: Starting Session 178489 of user deploy.  
Dec 14 11:40:53 iZ258fuhh93Z rsyslogd: \[origin software="rsyslogd" swVersion="7.4.7" x-pid="471" x-info="http://www.rsyslog.com"\] star

Dec 14 19:40:46 iZ258fuhh93Z journal: Runtime journal is using 8.0M (max allowed 189.5M, trying to leave 284.3M free of 1.8G available → current limit 189.5M).  
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpuset  
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpu  
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpuacct

>>>>>>> 重启

Dec 14 19:40:46 iZ258fuhh93Z kernel: Linux version 3.10.0-327.10.1.el7.x86\_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) \#1 SMP  
Tue Feb 16 17:03:50 UTC 2016  
Dec 14 19:40:46 iZ258fuhh93Z kernel: Command line: BOOT\_IMAGE=/boot/vmlinuz-3.10.0-327.10.1.el7.x86\_64 root=UUID=6634633e-001d-43ba-8fab-202f1df93339 ro crashkernel=auto vco  
nsole.font=latarcyrheb-sun16 vconsole.keymap=us biosdevname=0 rhgb quiet LANG=en\_US.UTF-8 net.ifnames=0  
Dec 14 19:40:46 iZ258fuhh93Z kernel: e820: BIOS-provided physical RAM map:  
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: \[mem 0x0000000000000000-0x000000000009fbff\] usable  
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: \[mem 0x000000000009fc00-0x000000000009ffff\] reserved  
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: \[mem 0x00000000000f0000-0x00000000000fffff\] reserved

分析日志发现主要问题是内存耗尽，无法创建ssh链接进程，可想，可能是什么服务耗尽了内存，导致服务器宕机

该问题很有可能和昨天晚上上的新任务有关系，要具体业务进行分析。

其次发现每次的cron任务里有发送邮件失败的问题，上网找到解决方案，解决它：

sendmail: fatal: parameter inet_interfaces: no local interface found for ::1

原因跟IPv6有关，解决办法就是让postfix只用IPv4。

打开文件：/etc/postfix/main.cf

把

inet_protocols = all

改成

inet_protocols = ipv4

完了之后执行

service postfix restart