Some Crucial Troubleshooting
In this section of the article Troubleshooting with Linux Logs, we will discuss some of the crucial troubleshooting examples using Linux logs. Here are the following:
Login Failures
If there are some users that have logged in or attempted to log in to your system, authentication logs can be checked that show the failure attempts. These authentication events are logged by the pluggable authentication module (PAM); if the authentication is successful, then the events will contain the strings such as “Accepted Password” and “Session Opened”. If the authentication gets failed, then the strings contained by the authentication events will be “Failed Password” and “User Unknown”.
To check the authentication failure message, the below command can be used:
sudo grep "authentication failure" /var/log/auth.log
For example, if the authentication is failed, then the failure events will look something like below:
pam_unixx(sshd:auth): authentication failure; logname= uid=2 euid=0 tty=ssh ruser= rhost=8.3.2.2
Failed password for invalid user NinjaXYZ from 10.0.2.2 port 5000 ssh4
pam_unix(sshd:auth): check pass; user unknown
PAM service(sshd) ignoringg max retrie; 4 > 3
Else If the authentication is successful, then successful events will look something like below:
Accepted password for NinjaXYZ from 8.3.2.2 port 5001 ssh4
pam_unix(sshd:session): session opened for user NinjaXYZ by (uid=2)
pam_unix(sshd:session): session closed for user NinjaXYZ
Memory Problems
A server might crash if there is a memory problem in the system, and the main problem can occur in memory when the RAM and swap space are completely exhausted. The kernel starts killing processes that are using the most memory. When the system uses, all the memory and a process tries to access the additional memory. Now what you can troubleshoot here is which process is killed instead of allowing that process to crash.
To check the process with the string “Out of memory” can be used as given in the below command:
sudo grep "Out of memory" /var/log/syslog
Here is an example output for the above showing that “process 16471” is being killed:
July 21 21:30:26 ip-172-31-34-37 kernel: [ 1575.404070] Out of memory: Kill process 16471 (NinjaXYZ) score 838 or sacrifice child
July 21 21:30:26 ip-172-31-34-37 kernel: [ 1575.408946] Killed process 16471 (NinjaXYZ) total-vm:144200240kB, anon-rss:562316kB, file-rss:0kB, shmem-rss:0kB
July 21 21:30:27 ip-172-31-34-37 kernel: [ 1575.518686] oom_reaper: reaped process 16471 (NinjaXYZ), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
System Crash
The problem can occur due to the system crash or reboot, and the method which can be used to display the logins for the “reboot” pseudo user. There is the command “last reboot”, which can be used here, as given below:
sudo last reboot
Here is an example output for the above showing the previous system reboots:
reboot system boot 5.4.0-80-generic Fri Aug 10 15:25 still running
reboot system boot 5.4.0-80-generic Thu Aug 12 09:10 still running
reboot system boot 5.4.0-80-generic Wed Aug 17 17:30 still running
reboot system boot 5.4.0-80-generic Tue Aug 19 14:55 still running
reboot system boot 5.4.0-80-generic Mon Aug 24 08:20 still running
Shutdown Concerns
If there are some problems and the administrator wants to see the log files, if someone ran the shutdown command manually. The below command can be used to check:
last | grep "ubuntu.*shutdown"
Here is an example output for the above showing if someone has used the shutdown command manually.
ubuntu pts/0 203.0.113.55 Sat Aug 21 14:30 - 14:35 (00:05) shutdown system down
ubuntu pts/0 203.0.113.55 Fri Aug 24 09:15 - 09:20 (00:05) shutdown system down
ubuntu pts/1 198.51.100.20 Thu Aug 29 16:40 - 16:45 (00:05) shutdown system down
Read about Linux System Logging and Troubleshooting.
Frequently Asked Questions
What is a logging system?
Logging is nothing but a facility provided by Linux by which users can track the information related to their system. There are different log files that basically contain the information which helps in troubleshooting issues.
What are the different types of logging systems?
There are different types of logs in Linux, such as System Logs, which contain information about the system’s operation. Application Logs contain information about the behavior of an application, and Service Logs contain information about all the services running on the system.
What services are provided by the logging system in Linux?
There are several services, such as the performance issues like memory leaks and disk input output bottlenecks that can be diagnosed, the system's health and also detection issues before they become critical, the logging system maintains the logs for compliance and auditing for many organizations.
How troubleshooting with Linux logs can be used for login failures?
If there are some users that have logged in or attempted to log in, authentication logs can be checked that show the failure attempts. To check the authentication failure message, the command can be used: sudo grep "authentication failure" /var/log/auth.log.
How memory problems can be resolved using Linux logs?
A problem can occur in memory when the RAM and swap space are completely exhausted. The kernel starts killing processes that are using the most memory. To check these processes, the following command can be used: sudo grep "Out of memory" /var/log/syslog.
Conclusion
There are different log files that basically contain the information which helps in troubleshooting issues. Logging is nothing but a facility provided by Linux by which users can track the information related to their system. In the article “Troubleshooting with Linux Logs”, we first discussed what is the logging system in Linux along with its services and its types. We also discussed some crucial troubleshooting that can be done using Linux logs.
Here are more articles that are recommended to read:
You can refer to our guided paths on the Coding Ninjas. You can check our course to learn more about DSA, DBMS, Competitive Programming, Python, Java, JavaScript, etc.
Also, check out some of the Guided Paths on topics such as Data Structure and Algorithms, Competitive Programming, Operating Systems, Computer Networks, DBMS, and System Design, etc. as well as some Contests, Test Series, Interview Bundles, and some Interview Experiences curated by top Industry Experts.
Happy Learning!
46+ registered 