If you have followed my posts here at Enable Sysadmin, you know that I previously worked as a storage support engineer. One of my many tasks in that role was to help customers replicate backups from their production environments to dedicated backup storage arrays. Many times, customers would contact me concerned about the speed of the data transfer from production to storage.
Now, if you have ever worked in support, you know that there can be many causes for a symptom. However, the throughput of a system can have huge implications for massive data transfers. If all is well, we are talking hours, if not... I have seen a single replication job take months.
We know that Linux is loaded full of helpful tools for all manner of issues. For input/output monitoring, we use the iostat command. iostat is a part of the sysstat package and is not loaded on all distributions by default.
Installation and base run
I am using Red Hat Enterprise Linux 8 here and have included the install output below.
[ Want to try out Red Hat Enterprise Linux? Download it now for free. ]
NOTE: the command runs automatically after installation.
[root@rhel ~]# iostat
bash: iostat: command not found...
Install package 'sysstat' to provide command 'iostat'? [N/y] y
* Waiting in queue...
The following packages have to be installed:
lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x86_64 Lm_sensors core libraries
sysstat-11.7.3-2.el8.x86_64 Collection of performance monitoring tools for Linux
Proceed with changes? [N/y] y
* Waiting in queue...
* Waiting for authentication...
* Waiting in queue...
* Downloading packages...
* Requesting data...
* Testing changes...
* Installing packages...
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
2.17 0.05 4.09 0.65 0.00 83.03
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 206.70 8014.01 1411.92 1224862 215798
sdc 0.69 20.39 0.00 3116 0
sdb 0.69 20.39 0.00 3116 0
dm-0 215.54 7917.78 1449.15 1210154 221488
dm-1 0.64 14.52 0.00 2220 0
If you run the base command without options, iostat displays CPU usage information. It also displays I/O stats for each partition on the system. The output includes totals, as well as per second values for both read and write operations. Also, note that the tps field is the total number of Transfers per second issued to a specific device.
The practical application is this: if you know what hardware is used, then you know what parameters it should be operating within. Once you combine this knowledge with the output of iostat, you can make changes to your system accordingly.
Interval runs
It can be useful in troubleshooting or data gathering phases to have a report run at a given interval. To do this, run the command with the interval (in seconds) at the end:
[root@rhel ~]# iostat -m 10
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.94 0.05 0.35 0.04 0.00 98.62
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 12.18 0.44 0.12 1212 323
sdc 0.04 0.00 0.00 3 0
sdb 0.04 0.00 0.00 3 0
dm-0 12.79 0.43 0.12 1197 329
dm-1 0.04 0.00 0.00 2 0
avg-cpu: %user %nice %system %iowait %steal %idle
0.24 0.00 0.15 0.00 0.00 99.61
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
dm-0 0.00 0.00 0.00 0 0
dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
0.20 0.00 0.18 0.00 0.00 99.62
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.50 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
dm-0 0.50 0.00 0.00 0 0
dm-1 0.00 0.00 0.00 0 0
The above output is from a 30-second run.
You must use Ctrl + C to exit the run.
Easy reading
To clean up the output and make it easier to digest, use the following options:
-m changes the output to megabytes, which is a bit easier to read and is usually better understood by customers or managers.
[root@rhel ~]# iostat -m
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.51 0.09 0.55 0.07 0.00 97.77
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 22.23 0.81 0.21 1211 322
sdc 0.07 0.00 0.00 3 0
sdb 0.07 0.00 0.00 3 0
dm-0 23.34 0.80 0.22 1197 328
dm-1 0.07 0.00 0.00 2 0
-p allows you to specify a particular device to focus in on. You can combine this option with the -m for a nice and tidy look at a particularly concerning device and its partitions.
[root@rhel ~]# iostat -m -p sda
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.19 0.07 0.45 0.06 0.00 98.24
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 17.27 0.63 0.17 1211 322
sda2 16.83 0.62 0.17 1202 320
sda1 0.10 0.00 0.00 7 2
Advanced stats
If the default values just aren't getting you the information you need, you can use the -x flag to view extended statistics:
[root@rhel ~]# iostat -m -p sda -x
Linux 4.18.0-193.1.2.el8_2.x86_64 (rhel.test) 06/17/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.06 0.06 0.40 0.05 0.00 98.43
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 12.20 2.83 0.54 0.14 0.02 0.92 0.16 24.64 0.55 0.50 0.00 45.58 52.37 0.46 0.69
sda2 12.10 2.54 0.54 0.14 0.02 0.92 0.16 26.64 0.55 0.47 0.00 45.60 57.88 0.47 0.68
sda1 0.08 0.01 0.00 0.00 0.00 0.00 0.00 23.53 0.44 1.00 0.00 43.00 161.08 0.57 0.00
Some of the options to pay attention to here are:
- avgqu-sz - average queue length of a request issued to the device
- await - average time for I/O requests issued to the device to be served (milliseconds)
- r_await - average time for read requests to be served (milliseconds)
- w_await - average time for write requests to be served (milliseconds)
There are other values present, but these are the ones to look out for.
Shutting down
This article covers just about everything you need to get started with iostat. If you have other questions or need further explanations of options, be sure to check out the man page or your preferred search engine. For other Linux tips and tricks, keep an eye on Enable Sysadmin!
[ Free online course: Red Hat Enterprise Linux technical overview. ]
