Subject: NFS on OSX Heads up info

From: TimY <[email-masked]>
Date: Tue, 12 Sep 2000 08:32:02 -0700

I just thought some people in this group might find this info useful 
and interesting.  Sorry if a bit off topic.  I don't want this 
posting to interrupt the netatalk info.

We have been testing NFS ver2 and ver3 on Mac OSX for about a month 
now.  We have really been hitting it hard.  And I mean really hard. 
Copying several gigs from several machines at one time 24 hours a day 
7 days a week for a month straight.  To our suprise it never crashed 
once.  We also tweaked with some settings on the OSX box (see below) 
I am including an email that sent to the OSX mailing list from my 
boss.  I think everyone will find it interesting.  Any questions you 
may have please feel free to email me.

ORIGINAL EMAIL TO OSX MAILING BEGINS HERE
==========================================
We have been running some test here as we plan to start using the NFS 
server of OS X Server under a good size load.  We used the NFS server 
way back in 1.0 but found it to crash the machine under heavy load. 
(a netinfo problem I belive.)  Things seem to be much more stable 
today.  We ran test for over a week without rebooting the server or 
noticing any major problems.  We did find a few strange things, and 
I'd be interested in what comments you all might have.

========
nbuf=
========
We started looking at what tuning avenues we could explore to make it 
as fast as possible.  A while ago, I tinkered around with the 
0200_Tune startup script.  From what I've heard, the 'nbuf=' boot 
argument increases the buffers used by the basic unix read and write 
commands.  When you change this value, the 'wired down' memory 
allocation changes.  My first guess would be that we might be able to 
see some nfsd performance changes with the size of this wired down 
buffer.  But, No.  It seems that this is not used at all.

========
nfsd -n
========
The -n option allows us to start several nfsd processes.  This didn't 
seem to effect things to much.  Our test machine has no other load so 
this might prove necessary once the machine has other stuff running 
on it.  The man page recommends one precess for each concurrent user. 
The default value of 6 gave us slightly better performance than some 
other values we tried.

========
ver2  || ver3  & proto=  (Client side setup, Solaris)
========
These flags on our nfs client (Solaris) proved to have a big effect. 
TCP protocol connections prove to be about 4-5 times faster than UDP. 
The version 2 or 3 didn't have a very big effect.  Version 2 looks 
like it might be just slightly faster than version 3.  We are still 
opting for version 3 with hopes that it will be improved in areas 
outside of our testing.

========
noac, actimeo=  (Client side setup, Solaris)
========
These flags have major impacts on read performance.  This will take 
some more testing and a better understanding on how caching might 
disturb our operation.  The noac option, that disables caching, took 
reads down to just a dribble.

========
pages free and inactive
========
This is very strange.  I'm only guessing as to what the term 
'inactive' means on this OS.  It looks related to cached io 
transactions such as mmap(2).  For example, accessing part or all off 
a large temp file allocates a bunch of memory as inactive.  If more 
active memory is needed for a processes, this memory is recycled.  If 
this file temp file is deleted, the inactive memory becomes free.

We noticed that this inactive memory would fluctuate wildly while we 
were testing nfsd.  We then wondered how bad would nfs performance 
get if most of the memory was either wired-down or active (not 
available to become inactive.)

We wired down as much memory as the OS would let us.  Then we opened 
lots of applications that would require a bunch of active memory 
without much cpu.  This left us with just a little memory that could 
be used for inactive.  To our surprise, nfs performance went up 
15-40% !!!  How could less memory mean better performance.  The only 
thing I can think of is that ram cache algorithm was creating extra 
unneeded transactions that was slower than our raw disk.

========
bottom line
========
The nbuf= and srv= boot arguments seem unrelated to the performance of nfsd

UDP sucks for NFS.  Use TCP instead.

Reducing client cache properties hurts performance, but might make 
for more accurate operation in a multi-client environment.

NFSD under MacOS X Server slows down with more available memory.
(Yikes! I can't belive I'm saying this!  Our testing was limited so 
this may not apply across the board.)

Please comment if you have any ideas on this.  My understanding of 
nbuf and inactive memory could be wrong.  Please explain if you know 
more.

========
testing setup
========
Server
Apple G3 beige tower G3 300, 66MHz bus, 256MB ram
Asante 10/100 Ethernet (DECchip 21140)
Apple LVD scsi with 9GB 10k lvd drive.
Mac OS X Server 1.2

Client
Dual PentiumPro 166 512k (clocked to 200), 320MB ram
Intel EtherExpressPro 10/100
Adaptec 2940UW2 with multiple drives
Solaris 7

Network
100Mb half-duplex shared hub.

Average results from iozone 100 (older version, single write then read):
ver2,udp,default memory:           1.0 MB/s write, 1.8 MB/s read
ver3,tcp,defualt memory:           4.2 MB/s write, 3.1 MB/s read
ver3,tcp,default memory,actimeo=0: 4.2 MB/s write, 0.8 MB/s read
ver3,tcp,low memory free:          5.8 MB/s write, 4.7 MB/s read

--