Sunday, August 14, 2011

Analyzing past System performance of a Linux server

Assumption: 
Today's date is 13th Aug, 2011.  You are asked to check the System performance of a Linux server on 7th Aug,2011 between 3 AM to 5 AM.

Solution: 
Run the 'sar' command on the respective 'sa' (System Activity) file created for the date "7th Aug,2011" with specifying the Starting and End time.

Illustration:
Go to /var/log/sa
[root@hostxyz sa]# ls -ltr sa??


-rw-r--r-- 1 root root 481776 Aug 5 23:50 sa05
-rw-r--r-- 1 root root 481776 Aug 6 23:50 sa06
-rw-r--r-- 1 root root 481776 Aug 7 23:50 sa07       # File that belongs to 7th Aug,2011
-rw-r--r-- 1 root root 481776 Aug 8 23:50 sa08
-rw-r--r-- 1 root root 481776 Aug 9 23:50 sa09
-rw-r--r-- 1 root root 481776 Aug 10 23:50 sa10
-rw-r--r-- 1 root root 481776 Aug 11 23:50 sa11
-rw-r--r-- 1 root root 481776 Aug 12 23:50 sa12
-rw-r--r-- 1 root root 287824 Aug 13 14:10 sa13
[root@hostxyz sa]#
[root@hostxyz sa]# sar -u -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01   # To check CPU utilization
Linux 2.6.18-92.el5 (hostxyz) 08/07/2011
03:00:01 AM CPU %user %nice %system %iowait %steal %idle
03:10:01 AM all 24.57 0.00 5.16 6.04 0.00 64.23
03:20:01 AM all 24.57 0.10 5.06 6.28 0.00 63.98
03:30:01 AM all 24.33 0.00 4.88 5.64 0.00 65.14
03:40:01 AM all 15.75 0.00 3.93 10.52 0.00 69.80
03:50:01 AM all 12.70 0.00 3.09 19.04 0.00 65.17
04:00:01 AM all 16.80 0.00 3.90 9.40 0.00 69.90
04:10:01 AM all 9.18 0.02 2.26 14.43 0.00 74.11
04:20:01 AM all 8.84 0.10 2.20 9.65 0.00 79.22
04:30:01 AM all 11.42 0.00 3.24 10.50 0.00 74.84
04:40:01 AM all 11.84 0.00 2.43 20.64 0.00 65.09
04:50:01 AM all 17.80 0.00 3.78 17.00 0.00 61.42
05:00:01 AM all 6.46 0.00 1.53 21.80 0.00 70.22
Average: all 15.35 0.02 3.46 12.58 0.00 68.59
[root@hostxyz sa]#
[root@hostxyz sa]#  sar -r -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Memory status


[Output no shown]
.
[root@hostxyz sa]#  sar -q -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Load average


[Output not shown]
.
[root@hostxyz sa]#  sar -b -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01     # To check I/O status
[Output not shown]


.
[root@hostxyz sa]#  sar -n DEV -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Network status


[Output not shown]
.
[root@hostxyz sa]# 


Notes: In Linux, System activity report is collected for every 10 minutes by a cron job "sysstat" located under /etc/cron.d and at end of the day, a summary report is generated and saved in /var/log/sa/saXX file, which we can use for later analysis. 

[root@hostxyz cron.d]# cat sysstat

# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A
root@hostxyz cron.d]#

8 comments:

  1. Nice presentation of SAR command.

    Thanks Ashok, For presenting the details of SAR. Try to update the monitoring tools.

    ReplyDelete
  2. I want to create file csv file dump the monthly server status reports by monthly. May i know whow can i do ?

    ReplyDelete
  3. Perl should be having some modules to convert the text file into CSV format.

    ReplyDelete
  4. How about learning KSar?

    ReplyDelete
  5. 'Ksar' seems to be a better one for doing System Analysis. I'm exploring how to use it. Thanks for letting me know.

    ReplyDelete
  6. Hi Ashok,
    Very Good solution you have provided.
    But I have some different question. Suppose a root filesystem has filled 90% so how I can get email or system message to know the current status?

    Thanks in advance.
    Arvind

    ReplyDelete