The network filesystem (NFS) allows machines to mount a disk partition on a remote machine as if it were a local disk. It allows for fast, seamless sharing of files across a network. However, because NFS relies on the existing network infrastructure, any glitches on the network may affect the performance of the connection. The two most important tools I have used over the years to analyze the performance of NFS from both the server and client perspective are nfsstat and nsfiostat. The two tools are part of the nfs-utils package and its needs to be installed as such:
yum install -y nfs-utils
Understanding the output of the tools can help with optimizing NFS performance.
The nfsstat command
The nfsstat command displays statistical information about the NFS and Remote Procedure Call (RPC) interfaces to the kernel.
On the NFS server, run the following command:
nfsstat -s
The output should be similar to the below:
Server rpc stats:
calls badcalls badclnt badauth xdrcall
107310012 0 0 0 0
Server nfs v4:
null compound
21 0% 107310004 99%
Server nfs v4 operations:
op0-unused op1-unused op2-future access close commit
0 0% 0 0% 0 0% 910848 0% 5671218 2% 153218 0%
create delegpurge delegreturn getattr getfh link
4104 0% 0 0% 517 0% 58659066 22% 8700995 3% 0 0%
The most important field to check is the badcalls, which represents the total number of calls rejected by the RPC layer. When the badcalls is greater than 0, than the underlying network needs to be checked, as there might be latency. Also, it's very important to put the NFS server in the same subnet as the NFS client when designing the network.
On the NFS client, you run the following:
nfsstat -c
Output should be similar to below:
Client rpc stats:
calls retrans authrefrsh
30557550 27686 11075
Client nfs v4:
null read write commit open open_conf
0 0% 1601136 5% 568239 1% 23683 0% 1652804 5% 1466260 4%
open_noat open_dgrd close setattr fsinfo renew
0 0% 0 0% 1648000 5% 27154 0% 8 0% 28320 0%
From the above, the client is doing well as it has relatively few retransmission requests. If you are encountering excessive retransmissions, you may want to adjust data transfer buffer sizes, which are specified by the mount command options rsize and wsize.
You can also check dropped packet by running the following command on both the server and the client:
nfsstat -o net
The nfsiostat command
The nfsiostat command works similarly to the iostat command, but is used for the NFS mount points on the server. It uses the file /proc/self/mountstats as input and provides information about the input/output performance of NFS shares mounted on the system.
The nfsiostat command is used on the NFS client to check its performance when communicating with the NFS server.
Running nfsiostat without any argument should have an output similar to the following:
10.10.1.10:/data/share mounted on /samba/students:
op/s rpc bklog
16.96 0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.900 5.392 5.990 0 (0.0%) 0.550 0.660
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.031 21.818 708.149 0 (0.0%) 122.745 35874.872
The most important statistics to look out for are:
retrans- This is the number of retransmissions.avg RTT (ms)- This is the duration from the time that the client's kernel sends the RPC request until the time it receives the reply.avg exe (ms)- This is the duration from the time that NFS client makes the RPC request to its kernel until the RPC request is completed. This includes the RTT.
A very high avg RTT(ms) and retrans indicates there is high latency on the network. High latency also affects the I/O performance, because there will be many timeouts. The client will become sluggish and difficult to work with.
It is also very important to always have a dedicated network between the NFS server and the NFS client to ensure high throughput.
Wrapping up
The use of nfsstat and nfsiostat to troubleshoot NFS performance issues can make you a much more efficient system administrator. Take some time to learn and use these great NFS tools.
[ Want more for your network? Download a free ebook on network automation with Ansible. ]
