I/O reporting from the Linux command line

If you have followed my posts here at Enable Sysadmin, you know that I previously worked as a storage support engineer. One of my many tasks in that role was to help customers replicate backups from their production environments to dedicated backup storage arrays. Many times, customers would contact me concerned about the speed of the data transfer from production to storage.

Now, if you have ever worked in support, you know that there can be many causes for a symptom. However, the throughput of a system can have huge implications for massive data transfers. If all is well, we are talking hours, if not... I have seen a single replication job take months.

We know that Linux is loaded full of helpful tools for all manner of issues. For input/output monitoring, we use the iostat command. iostat is a part of the sysstat package and is not loaded on all distributions by default.

Installation and base run

I am using Red Hat Enterprise Linux 8 here and have included the install output below.

[ Want to try out Red Hat Enterprise Linux? Download it now for free. ]

NOTE: the command runs automatically after installation.

[root@rhel ~]# iostat
bash: iostat: command not found...
Install package 'sysstat' to provide command 'iostat'? [N/y] y
    
    
 * Waiting in queue... 
The following packages have to be installed:
lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x86_64    Lm_sensors core libraries
sysstat-11.7.3-2.el8.x86_64    Collection of performance monitoring tools for Linux
Proceed with changes? [N/y] y
    
    
 * Waiting in queue... 
 * Waiting for authentication... 
 * Waiting in queue... 
 * Downloading packages... 
 * Requesting data... 
 * Testing changes... 
 * Installing packages... 
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.17    0.05    4.09    0.65    0.00   83.03
    
Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             206.70      8014.01      1411.92    1224862     215798
sdc               0.69        20.39         0.00       3116          0
sdb               0.69        20.39         0.00       3116          0
dm-0            215.54      7917.78      1449.15    1210154     221488
dm-1              0.64        14.52         0.00       2220          0

If you run the base command without options, iostat displays CPU usage information. It also displays I/O stats for each partition on the system. The output includes totals, as well as per second values for both read and write operations. Also, note that the tps field is the total number of Transfers per second issued to a specific device.

The practical application is this: if you know what hardware is used, then you know what parameters it should be operating within. Once you combine this knowledge with the output of iostat, you can make changes to your system accordingly.

Interval runs

It can be useful in troubleshooting or data gathering phases to have a report run at a given interval. To do this, run the command with the interval (in seconds) at the end:

[root@rhel ~]# iostat -m 10
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.94    0.05    0.35    0.04    0.00   98.62
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              12.18         0.44         0.12       1212        323
sdc               0.04         0.00         0.00          3          0
sdb               0.04         0.00         0.00          3          0
dm-0             12.79         0.43         0.12       1197        329
dm-1              0.04         0.00         0.00          2          0
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.24    0.00    0.15    0.00    0.00   99.61
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.20    0.00    0.18    0.00    0.00   99.62
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.50         0.00         0.00          0          0
sdc               0.00         0.00         0.00          0          0
sdb               0.00         0.00         0.00          0          0
dm-0              0.50         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0

The above output is from a 30-second run.

You must use Ctrl + C to exit the run.

Easy reading

To clean up the output and make it easier to digest, use the following options:

-m changes the output to megabytes, which is a bit easier to read and is usually better understood by customers or managers.

[root@rhel ~]# iostat -m
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.51    0.09    0.55    0.07    0.00   97.77
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              22.23         0.81         0.21       1211        322
sdc               0.07         0.00         0.00          3          0
sdb               0.07         0.00         0.00          3          0
dm-0             23.34         0.80         0.22       1197        328
dm-1              0.07         0.00         0.00          2          0

-p allows you to specify a particular device to focus in on. You can combine this option with the -m for a nice and tidy look at a particularly concerning device and its partitions.

[root@rhel ~]# iostat -m -p sda
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.19    0.07    0.45    0.06    0.00   98.24
    
Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              17.27         0.63         0.17       1211        322
sda2             16.83         0.62         0.17       1202        320
sda1              0.10         0.00         0.00          7          2

Advanced stats

If the default values just aren't getting you the information you need, you can use the -x flag to view extended statistics:

[root@rhel ~]# iostat -m -p sda -x 
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test)     06/17/2020     _x86_64_    (4 CPU)
    
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.06    0.06    0.40    0.05    0.00   98.43
    
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda             12.20    2.83      0.54      0.14     0.02     0.92   0.16  24.64    0.55    0.50   0.00    45.58    52.37   0.46   0.69
sda2            12.10    2.54      0.54      0.14     0.02     0.92   0.16  26.64    0.55    0.47   0.00    45.60    57.88   0.47   0.68
sda1             0.08    0.01      0.00      0.00     0.00     0.00   0.00  23.53    0.44    1.00   0.00    43.00   161.08   0.57   0.00

Some of the options to pay attention to here are:

• avgqu-sz - average queue length of a request issued to the device
• await - average time for I/O requests issued to the device to be served (milliseconds)
• r_await - average time for read requests to be served (milliseconds)
• w_await - average time for write requests to be served (milliseconds)

There are other values present, but these are the ones to look out for.

Shutting down

This article covers just about everything you need to get started with iostat. If you have other questions or need further explanations of options, be sure to check out the man page or your preferred search engine. For other Linux tips and tricks, keep an eye on Enable Sysadmin!

[ Free online course: Red Hat Enterprise Linux technical overview. ]