nfs write problems

[Les]

On Tue, 2007-10-02 at 15:56 -0400, Bob Kryger wrote:
> So, I have a relatively new system on which I am seeing strange NFS 
> behavior.
> 
> In short I am getting seemingly random errors in files written via NFS.
> 
>     * I do not get the errors if I write files locally.
>     * I have no errors in the NIC, I even tried a second NIC in a PCI
>       slot as opposed to the onboard one. There are no errors recorded
>       on the NIC or the switch on a 1Gb port.
>     * I see no memory errors, I ran memtest for 3 days clean.
>     * To test I am using dd if=/dev/zero of various (large) file sizes.
>     * Since I know that the file should be all zeros I wrote a C program
>       to read it back and tell me where it finds non-zero bytes. The
>       program results are confirmed with od.
>     * The files read back always have the errors in the same place, so
>       it is not a problem with reading the files.
>     * There are no errors in any logs.
>     * The problem occurs on both the RAID1 (ext3) and RAID10 (xfs)
>       filesystems.
>     * I've tried two clients, both FC5 one 64bit, and the other a 32 bit
>       with the same results. This error was uncovered by users
>       attempting to write files from other systems and other Fedora
>       releases, so it is repeatable regardless of the client.
>     * the server is not running anything else and spends a large portion
>       of the time idle. loadaverages are quite low. swap is mostly
>       unused. a large portion of RAM is allocated to file cache, but I
>       expect that this would be normal for this amount of file IO.
> 
> The server is running an up-to-date FC6, although this also occurred 
> with FC5. I am about to try F7.
> 
> Hardware is an AMD 1220 dual core 64bit, on a Tyan K8SSA S3950
> with an Adaptec Raid 2230SLP and 7 Fujitsu MAU3147NC.
> The RAID config is that 1 disks (on diff channels) are in a Mirror for 
> the OS, 4 are in a Raid 10 config and 1 is a hot spare.
> 
> Anyone ever seen anything like this before?
> Suggest where I might look next?
> Additional tests?
> 
Ox0D is suspecious.  This is the carriage return character.  How did you
create the file?  Some forms of write, and some utilities insert end of
line characters, so this value might be an attribute of that effect.

Regards,
Les H