Using nfsstat and nfsiostat to troubleshoot NFS performance issues on Linux

The network filesystem (NFS) allows machines to mount a disk partition on a remote machine as if it were a local disk. It allows for fast, seamless sharing of files across a network. However, because NFS relies on the existing network infrastructure, any glitches on the network may affect the performance of the connection. The two most important tools I have used over the years to analyze the performance of NFS from both the server and client perspective are nfsstat and nsfiostat. The two tools are part of the nfs-utils package and its needs to be installed as such: 

yum install -y nfs-utils

Understanding the output of the tools can help with optimizing NFS performance.

The nfsstat command

The nfsstat command displays statistical information about the NFS and Remote Procedure Call (RPC) interfaces to the kernel.

On the NFS server, run the following command:

nfsstat -s

The output should be similar to the below:

Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
107310012   0          0          0          0

Server nfs v4:
null         compound
21        0% 107310004 99%

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit
0         0% 0         0% 0         0% 910848    0% 5671218   2% 153218    0%
create       delegpurge   delegreturn  getattr      getfh        link
4104      0% 0         0% 517       0% 58659066 22% 8700995   3% 0         0%

The most important field to check is the badcalls, which represents the total number of calls rejected by the RPC layer. When the badcalls is greater than 0, than the underlying network needs to be checked, as there might be latency. Also, it's very important to put the NFS server in the same subnet as the NFS client when designing the network.

On the NFS client, you run the following:

nfsstat -c

Output should be similar to below:

Client rpc stats:
calls      retrans    authrefrsh
30557550   27686      11075

Client nfs v4:
null         read         write        commit       open         open_conf
0         0% 1601136   5% 568239    1% 23683     0% 1652804   5% 1466260   4%
open_noat    open_dgrd    close        setattr      fsinfo       renew
0         0% 0         0% 1648000   5% 27154     0% 8         0% 28320     0%

From the above, the client is doing well as it has relatively few retransmission requests. If you are encountering excessive retransmissions, you may want to adjust data transfer buffer sizes, which are specified by the mount command options rsize and wsize.

You can also check dropped packet by running the following command on both the server and the client:

nfsstat -o net

The nfsiostat command

The nfsiostat command works similarly to the iostat command, but is used for the NFS mount points on the server. It uses the file /proc/self/mountstats as input and provides information about the input/output performance of NFS shares mounted on the system.

The nfsiostat command is used on the NFS client to check its performance when communicating with the NFS server.

Running nfsiostat without any argument should have an output similar to the following:

10.10.1.10:/data/share mounted on /samba/students:
   op/s         rpc bklog
  16.96            0.00
read:             ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                  0.900           5.392           5.990        0 (0.0%)           0.550           0.660
write:            ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                  0.031          21.818         708.149        0 (0.0%)         122.745         35874.872

The most important statistics to look out for are:

• retrans - This is the number of retransmissions.
• avg RTT (ms) - This is the duration from the time that the client's kernel sends the RPC request until the time it receives the reply.
• avg exe (ms) - This is the duration from the time that NFS client makes the RPC request to its kernel until the RPC request is completed. This includes the RTT.

A very high avg RTT(ms) and retrans indicates there is high latency on the network. High latency also affects the I/O performance, because there will be many timeouts. The client will become sluggish and difficult to work with.

It is also very important to always have a dedicated network between the NFS server and the NFS client to ensure high throughput.

Wrapping up

The use of nfsstat and nfsiostat to troubleshoot NFS performance issues can make you a much more efficient system administrator. Take some time to learn and use these great NFS tools.

[ Want more for your network? Download a free ebook on network automation with Ansible. ]