Ideally one should everyday login to Ubuntu server running WordPress like software. Public internet has been dangerous and hackers known to perform various kind of practice. It is not great get astonished by an unexpected shutdown of cloud server in clueless manner.
Overheating a common issue for the dedicated servers while cloud Server instances commonly have the issue of operating system related failure of “forced upgrade” resulting dumb, mute server. Many start with Ubuntu as they feel it is easy but time push everyone towards becoming semi-expert and to shift to REHL or CentOS like production grade server operating systems.
Programs root privileged programs can gracefully shutdown an operating system. Either a humane user with root privileges or an acpi script can do that job. In either cases we can find out the reason by checking the logs. Overheating a common issue for the dedicated servers. An acpi shutdown can be caused by power button press, overheating. Virtual and cloud have virtual buttons on control panel. Someone can login to your control panel to shut down the instance.
---
Not always system have the chance to save logs. But monitoring logs, graphs of the system will show load average charting, other matters which can indicate the scence.
How To Examine Unexpected Shutdown of Cloud Server From SSH
You must SSH as fast you detected such scene. Hackers love to add scripts via crontab, so check it :
1 | sudo crontab -e |
Usually /var/log will tell the time the machine was shut down but not the initial reason, it will be like :
1 | shutdown[25861]: shutting down for system halt |
Command can be like :
1 2 3 | grep -iv ': starting\|kernel: .*: Power Button\|watching system buttons\|Stopped Cleaning Up\|Started Crash recovery kernel' \ /var/log/messages /var/log/syslog /var/log/apcupsd* \ | grep -iw 'recover[a-z]*\|power[a-z]*\|shut[a-z ]*down\|rsyslogd\|ups' |
Check the following log files with the keyword shutdown :
1 2 3 4 5 | /var/log/debug /var/log/syslog (will be pretty full and may be harder to browse) /var/log/user.log /var/log/kern.log /var/log/boot |
Of course there are commands to check :
Display list of last reboot entries:
1 | last reboot | less |
Display list of last shutdown entries:
1 | last -x | less |
or
1 2 3 4 5 6 7 8 9 10 11 12 | last -x | grep shutdown | less # last -x | head | tac # last reboot | head -6 |
This is a normal shutdown :
1 2 3 4 | runlevel (to lvl 0) 2.6.32- Sat Mar 02 09:48 - 08:51 (00:02) shutdown system down ... <-- first the system shuts down reboot system boot ... <-- afterwards the system boots runlevel (to lvl 3) |
This kind of keywords in log :
1 | rsyslogd: ... exiting on signal 15 |
probably some software performed orderly shutdown. Run these :
1 2 | systemd-analyze blame system-analyze blame |
Rsyslog hanged this server inviting shutdown :
Removing it will fix the problem as emergency measure. Why it acted oddly requires research. Hacker’s altered script can fake odd behavior. Is such hanging by rsyslog happened to the others? Web search saying yes. So, it was probably not out of intrusion.
There is a good old article on this topic :
1 | https://web.archive.org/web/20170904210829/http://scofaq.aplawrence.com:80/FAQ_scotec1haltcatch.html |
We hope, Ubuntu users conclude reason of shutdown to this (and not an intrusion) :
1 | https://askubuntu.com/questions/878630/apt-unattended-upgrades-stalls-shutdown |
You can talk with your host. The thing is not good if repeats twice. Result with no backup can be sucking thumb.
Tagged With https://thecustomizewindows com/2019/03/analyzing-unexpected-shutdown-of-ubuntu-cloud-server/