gistfile1.txt

on linux (CentOS 4 or 5) how can I control client buffering?  I need
the client to cache writes longer.  I have an app that does 100s of
thousands of 1 to 8 KB writes.  When I wrote a C program to do that,
I get 50 MB /sec (1024 writes of 1024 bytes).  The application however
(proprietary, similar to an app based on say Berkeley DB) gets 4 MB/sec.
if I move the app onto the file server and run it I get about 40 MB/sec,
sustained, so there seems to be no problem with the file system involved.

If I put fsync between my 1 K writes, I get about 0.1 MB sec (100 writes
per second roughly).  10 times slower than the third party app.

I used strace to verify what the third party app is doing.  in 3 seconds,
the app does a total of 13 MB (4 MB/sec instead of the 1 MB/sec I
stated above)

retransmits is about 1 retransmit for 20,000 requests.

client1 settings:

alpha:/mnt/tera/appl on /appl type nfs (rw,noatime,proto=udp,soft,bg,intr,addr=10.1.10.51)

client2 settings (from fstab as it is currently unmounted):

10.1.10.51:/mnt/tera/appl /appl nfs rw,async,vers=3,rsize=32768,wsize=32768,soft,intr,bg,proto=tcp,timeo=600,retrans=2,sec=sys,noatime

No appreciable performance difference between the two (tcp vs udp).

Changing rsize and rsize didn't seem to change anything.

Server has 4 cores and 32 threads.  the move from 8 to 32 threads did not
improve performance (not expected to as the process making the file IO is a
single thread of execution.

Here is a 3 second strace.  I removed the data written and read both
for client data privacy reasons and for readability.  The same regex
also removed the lseek offset, but you get the idea.

In about 3 seconds:

4237 lseeks
3780 reads
468 writes

files not opened with O_SYNC
no calls to sync() of fsync()

IO total 13034433 bytes

http://gist.github.com/130287 is the strace data