NFS Performance

I benchmarked the NFS performance of recent versions (1.3.62, 1.3.67p1)
of the Linux kernel to quantify the effect of adding NFS caching. I also
compared the results to FreeBSD because it's considered to have
a high performance NFS implementation.

I used the bonnie benchmark to measure raw throughput. I ran bonnie
using a 2 MB and a 10MB file across NFS and to the local disk,
respectively. I used the benchmark listed below to measure attribute
reading (stat()) performance. I ran it on the Red Hat Distribution's
standard /etc directory.

	time find . -print -exec ls -l {} \; > /dev/null

I used three 100 Mhz Pentiums each with 32 MB of memory and 10 Mb/sec
Ethernet cards. Both of the server machines were running Linux 1.3.67.
"tnt" is the client machine in all cases. "c4" is a server on the same
physical subnet as "tnt". "semtex" is a server one hop away from "tnt".
The router is supposedly made by Cisco and uses switched Ethernet.

The following results are for Linux 1.3.62:

	tnt->c4		tnt->semtex	tnt (locally)
bonnie:	(all results are in KB/sec.)
write:	646 		466		10290
read:	628		454		17558

find:	13.42 sec.	14.68 sec.	12.26 sec.
	
This version of the Linux kernel is able to read/write cached files
which are destined for the local disk at very high bandwidths. It
is not able to cache files accessed across NFS. The router decreases
bandwidth to 2/3 of the the normal NFS bandwidths. Other investigation
reveals that the is little traffic on either subnet, so collisions
on the physical network are not causing this poor performance.

NFS adds little additional penalty to the find benchmark.

The following results are for Linux 1.3.67:

	tnt->c4		tnt->semtex	tnt (locally)
bonnie:	(all results are in KB/sec.)
write:	636 		440		7392
read:	20159		20150		20172

find:	13.71 sec.	15 sec.		11.24 sec.

1.3.67 is able to cache NFS reads. This increases read bandwidth for
cached reads by more than 30 times. When I ran this benchmark on a
client machine with a faster chip set and memory, the read bandwidth
increased to more than 50 times faster than uncached reads, so this is
primarily a function of memory bandwidth (as would be expected).

Writes are not buffered. The server writes to disk asynchronously
anyway, so forcing the client to write synchronously to the server
increases reliability only if the network is less reliable than the
server.

The router does not affect cached read performance, but still decreases
write performance by 1/3.

The NFS caching does not affect file attribute reading performance.
Although NFS adds little penalty to file attribute reading performance
in the first place, one would expect that the running time for the
find benchmark across NFS would match the time when using the local
disk. This difference is either the result of experimental error or
the attributes are not being cached like the data.

The following results are for FreeBSD 2.0.5:

	tnt->c4		tnt->semtex	tnt (locally)
bonnie:	(all results are in KB/sec.)
write:	815 		521		5157
read:	17728		17481		15934

find:	7.33 sec.	no result	4.2 sec.

For cached reads, Linux 1.3.67 achieves a higher read bandwidth than
FreeBSD.

Like Linux, FreeBSD does not buffer NFS writes at the client, but it
does achieve a higher unbuffered write bandwidth than Linux. This is
possible because of its more efficient networking implementation and
possibly because it implements NFS in the kernel instead of as a user
level daemon like Linux does.

When the files are on the local disk, FreeBSD is able to read file
attributes 3 times faster than Linux. When the files are accessed
across NFS, FreeBSD is able to read file attributes 2 times faster
than Linux. FreeBSD haa a separate cache for the file attributes so
that they will not be flushed from memory by file data. Given the
smaller size of the attribute cache and the specialized data it
contains, it can probably be accessed faster than the buffer cache.
Linux's poor performance on this benchmark suggests that is does not
have such a cache.

In summary, Linux now has much faster NFS read performance, but still
lags behind FreeBSD in write performance. For a workstation cluster
that mainly uses NFS to share program binaries, this shouldn't be a
problem. It would be a problem for diskless workstations.

Linux still reads file attributes much more slowly than FreeBSD from
the local disk and across NFS. Certain applications query file
attributes frequently and their performance is bound by how quickly
the system can read file attributes. Examples are any application that
uses the file system as a database, like a news server or CVS. Using
"make" on the a large source tree (like the Linux kernel) is another
example. "make" will spend a long time stat()-ing files even if few of
them need to be recompiled.