Subject: Re: Locking propigation probelm samba to netatalk & netatalk to samba

From: Mike Brodbelt <[email-masked]>
Date: Wed, 18 Oct 2000 11:22:14 +0100

Mike Fedyk wrote:
> 
> Take a look at this output from lsof:
> smbd    11490        0    8u   REG    3,3 55296 1136647 Maintenanc.fp5
> afpd    31010        0    2u   REG    3,3  3207 1136655 auction - envelope.rtf
> afpd    31010        0    5u   REG    3,3  4156 1136656 Word Work File L 1
> afpd    31010        0    8u   REG    3,3 55296 1136647 Maintenanc.fp5
> smbd    31386        0    9u   REG    3,3  4156 1136656 Word Work File L 1
> 
> This is showing a rtf file open in word on mac and windows, both writable.  I am
> able to save on one while the other has the same file open.  The information
> isn't transferring between samba and netatalk.  Locking works within samba and
> within netatalk, meaning mac-mac locking conflicts are reported and smb-smb
> locking conflicts are too.  But if you have the same file open in windows and
> mac, there isn't any warning!

As I understand it, Samba 2.0.7 implements oplocks for windows clients,
but those locks do not propagate to the underlying Unix system. This can
cause locking problems with Unix apps and windows apps accessing the
same files, which is essentially what you are seeing. Neither Netatalk
or Samba have any idea that the file is locked by the other.

One of the new features of Samba 2.2 is the mapping of oplocks to POSIX
locks. From the release notes:-

========================================================
Rewritten internal locking semantics for more robustness.
This alpha supports full 64 bit locking semantics on all
(even 32 bit) platforms. SMB locks are mapped onto POSIX
locks (32 bit or 64 bit) as the underlying system allows.
========================================================

This should mean that, assuming your undelying system supports POSIX
locks, then Samba will have done its bit with regard to this. You still
need Netatalk to honour the POSIX locks, and also to map Mac locks to
POSIX locks to be able to use this sort of setup with impunity. I have
no idea what the state of this sort of thing is within Netatalk - maybe
someone on the Netatalk list will know this...

I'd suggest that you make Mac and Windows users use different shares for
write access. Allow both groups read-only access to the others shares,
and if they need to edit a file, they can make a copy. Otherwise, you're
probably in for large amounts of pain....

HTH

Mike.

-- 
Unsubscribe?  mail -s unsubscribe [email-masked] < /dev/null

---

From: Mike Fedyk <[email-masked]>
Date: Wed, 18 Oct 2000 01:12:00 -0700

I've turned off oplocks, level 2 oplocks, kernel oplocks and turned on strict
locking.  Same difference.

This is a Debian 2.2 box running 2.2.17  I am attaching my netatalk and samba
configs.

Please help!

I'm getting corrupted files because of this. :(
-- 

Mike Fedyk                   "They that can give up essential liberty
Information Systems           to obtain a little temporary safety
Match Mail Productions Inc.   deserve neither liberty nor safety."
[email-masked]                                   Ben Franklin

---

From: "Daniel E. Lautenschleger" <[email-masked]>
Date: Tue, 13 Mar 2001 09:07:22 -0600 (CST)

This is the type of information many of us have been looking for with
respect to Netatalk and Samba coexisting in a "friendly" matter. Many of
the replies I received when originally asking whether the two behaved
together basically stated that the biggest problem was a Windows user and
a Netatalk user accessing the same file at the same time.

Some of the other comments regarding Samba and Netatalk being used on the
same share...

Sorry about the long post, but this is pretty important to a lot of
people.

-----

I've been doing this for awhile.  I have a stable netatalk/samba server
at netatalk 1.4b2+asun.  No trouble; unless my users aren't telling me.
Read the mailing lists - there a --with-netatalk config time option (not
in the build docs that I could see, but it was covered on the mailing
list IIRC).  --with-netatalk is currently an experimental hack, if I
remember, it forces samba to move the .AppleDouble entries if a PC user
moves files around from the PC (so as not to pooch appleshare side
stuff).  Don't know how reliable it is, I only included that option on
the most recent server I config'd and it hasn't been tested thoroughly
yet.

-----

I've been researching this for a while, and the biggest problem I've seen
is incompatible locking.  Simultaneously opening a datafile from both a
Samba connected PC and a Netatalk connected Mac can cause strange things
to
happen if they both try to modify the file.  There are no errors, and the
file can easily be corrupted.

AFAIK the --with-netatalk in Samba does nothing.  Unless something has
been
changed recently, it was a feature that was never finished.

----

The two seem to co-exist together pretty well.  Rich made a
few valid points with .AppleDouble files.

        However, there is one thing that you may want to bring up in
regards to Windows and Linux users.  You need to let them know that
they can't create files greater than 32 characters or the Mac users
won't be a able to see them.  And you will get holy hell from your
Mac users when they call you and day "Bob said that he put the file
on the server, but I can't see it, is the server broken?"
        So, you may need to do some social engineering with your
Windows and Linux users to make sure that they understand the
limitation.  They will forget about it anyway, so make sure you have
them sign something.  :-)  Can you tell I've been an Admin for way
too long?

----

There is some sort of enable-netatalk-compatibility (or something similar
to that) option in Samba which is supposed to, I repeat *supposed to* make
it easier for netatalk to understand the file types.  But I've never tried
it and I've heard some say that it doesn't work and they can't figure out
what that feature is supposed to be for.  Might be worth investigating,
though.

----

On the other hand I had the oportunity to look pretty deep into samba's
mechanisms and I never heard or read about any  "netatalk compatibility"
option though. So personnaly I would say go for it, but make sure of two
things to avoid mac users coming at you complaining about generic icons
and
apps not launching:

- first have your .AppleVolumes.system up to date so files coming from a
PC
(and without resources fork) are seen properly on macs. The
.AppleVolume.system does that by "telling" files to assume such or such
mac-identity according to their file extension.

-second, you will have to somehow get the mac users into the habit of
naming
their files with an extension, so that they are seen properly on windows
stations. And this is not easy, let me tell you that!

----

Make sure that you're running a version of netatalk that has working
codepage support (the current snapshots are broken in this regard, AFAIK)
if you want your users to use German umlauts and such in their
filenames. Then you need to configure both Samba and Netatalk to use the
same codepage. Current versions also have a mswindows options which
disallows the creation of filenames containing the characters

/\:*?"<>|

which the PC can't deal with (my PC actually crashes when trying to
access a file containing ':' that was created on the Mac).

I'm still looking for a version of Netatalk that has _both_ working
codepage support and a working mswindows option . . . which would really
resolve all the problems we've had.

The locking issue is another matter (see below in this thread).
Interestingly, we haven't come across a single incident of this being
a problem in about 6 months of running Netatalk/Samba in a Network with 10
Macs and 5 PCs.

-----

Albeit better than the NON-EXISTANCE of shared locking between
Samba/Netatalk (although the new Samba has a locking API, so
theoretically any program can use its locking API to "stay safe" with
Samba), NT/2k still do not lock well between the two platforms. There is
no way to lock parts of a file,  and write-locks don't exist, so both
plats can simultaneously write to the same file and possibly corrupt it
horribly.

----

To make Samba and Netatalk work together, it is a good plan to veto
those files which are automagically created by Netatalk on the Samba
side. My veto string looks like this:

/.AppleDesktop/.AppleDouble/Network Trash
Folder/TheVolumeSettingsFolder/TheFindByContentFolder/Temporary
Items/lost+found/Folder Settings/.finf/

What I have not been able to figure out is how to veto Icon^M files.
Does anybody know?

----

I wonder if collaboration between Netatalk folks and Samba folks could
occur?!

God, it would be sweet to have both coexist, eh!

---

From: "Kevin M. Myer" <[email-masked]>
Date: Tue, 13 Mar 2001 15:56:23 -0500 (EST)

> Some of the other comments regarding Samba and Netatalk being used on the

Sorry about all the CC's: on this message.

I posted on this subject awhile back, the subject was something like "The
facts about Samba and netatalk" - search the archives.

To summarize:

the --with-netatalk option in Samba does _NOTHING_, except enable a few
#ifdefs in the samba code, which do....nothing.  There was experimental
wrapper functions for .AppleDouble support written up in 1998 but the
hooks were never placed in the runtime code to actually use these
functions.  There's no code whatsoever written to deal with cooperative
locking and as far as I know, netatalk and samba use incompatible lock
types. Also, as far as I know, the only real degree of integration with
Samba is to specify, in the smb.conf file, to block the viewing of
netatalk metafiles (.AppleDouble, etc.) and to allow deletion of the
netatalk meta files when a directory is being deleted (otherwise the
delete will return a directory not empty errno).

I requested that the samba folks remove the --with-netatalk config option
and the unused libatalk.c code since its only causing confusion and
propagation of misinformation over mailing lists (I myself thought it was
actually doing things as well for awhile, until I ran some tests and
checked the source code).  I haven't checked the code lately to see if
this was actually accomplished.

There was no interest from the samba folks to impliment this, judging from
the zero replies I got regarding updating the netatalk support.  I floated
a question to see if anyone on the netatalk list was interested in it and
got a few responses from others who had posed the same question with
generally the same ideas but nothing ever materialized from their posts or
from mine.

So I think for the time being, unless someone changes the locking
mechanism used with netatalk, its safe to say that using netatalk and
samba together is a calculated risk, i.e. you run the risk that a Mac and
a Windows machine will have open the same file and can then easily corrupt
it at worse or whoever saves last will get their changes saved at best.

Incidentally, I also reported a problem with the --enable-flock-locks
configure option with the 1.5preX series.  Samba aside, just enabling that
broke netatalk's locking mechanism, at least under kernel 2.2.X on Linux.
I never got a confirmation that this was a widespread problem, although it
occured on three machines I tested here.  However, I noticed that its
commented out in the latest 1.5-pre5 RPM .spec file so

> God, it would be sweet to have both coexist, eh!

Yes, it would be.  Have samba be aware of netatalk meta-files and have
netatalk use samba's locking mechanism.  That would equal a fully
cooperative SMB and AppleShare file share volume on a robust operating
system, something that currently can't be said for NT and Services for
Macintosh.

If anyone has an update on anything I've listed, I'd really like to hear
about it (i.e. if you are working on implimenting samba/netatalk
interoperability or if you have more info on locking problems).

Thanks,
Kevin

-- 
Kevin M. Myer
Systems Administrator
Lancaster-Lebanon Intermediate Unit 13
(717)-560-6140

---

From: Frank Joerdens <[email-masked]>
Date: Tue, 13 Mar 2001 22:17:59 +0100

On Tue, Mar 13, 2001 at 03:56:23PM -0500, Kevin M. Myer wrote:
[ . . . ]
> interoperability or if you have more info on locking problems).

I don't have more info on the locking issue except that I find it rather
remarkable that it has not caused _any_ problem whatsoever in about 6
months+ of operation in a Samba/Netatalk environment that's fairly
heavily used (about a dozen macs and a handful of PCs accessing one
Linux box with about 100 GB shared space). This means that from my
experience at least, the locking issue is only the worst problem in
theory. My pet peeves are codepages and mswindows options not working
together (a Netatalk issue), and vetoing with certain files that contain
metacharacters such as '^M' or '?' (a Samba issue), e.g. Icon files that
are somethimes created with certain files by Netatalk and which are
still visible to the PCs.

What I'm wondering is: Isn't the whole issue becoming somewhat moot with
Mac OS X which has NFS support, and a flat filesystem to boot? 

Regards, Frank

---

From: Hauke Fath <[email-masked]>
Date: Wed, 14 Mar 2001 12:18:53 +0100

>Mac OS X which has NFS support, and a flat filesystem to boot?

Just about when the last "Classic" MacOS machine leaves the shop. 
I.e.: "Not yet."

And, while NFS may be supported by X, I have not come across a 
statement that it is going to be the preferred filesharing mechanism.

	hauke

-- 
Hauke Fath                        tangro software components GmbH
                                         D-69115 Heidelberg
[email-masked]                       Ruf +49-6221-13336-0, Fax -21

---

From: testing roaming <[email-masked]>
Date: Wed, 14 Mar 2001 10:54:48 -0500

Hauke,

	I've been living with various appleshare servers for more
than a decade -- implementations from Pacer software on VMS (gasp)
and Unix, CAP, Netatalk.   Several vendors attempted implementations
of what they called "NFS/Share" or something like that.   All, as
I recall, were horrendously slow -- not for file transfer but for
duirectory scanning.   This was explained to me as having
to do with the way in which Appleshare get's to know what is in
subdirectories and required single iterative calls to find out what
was there -- very bad if your directory has a few thousand files.
I don't know if this is a technical issue that can't be overcome,
but it seemed not to be trivial.
	I believe they all disappeared.  Are any still around that 
you know of?

						Matthew Temple

-- 
=============================================================
Matthew Temple                Tel:    617/632-2597
Director, Research Computing  Fax:    617/632-4012
Dana-Farber Cancer Institute  [email-masked]
44 Binney Street, Smith 345   http://research.dfci.harvard.edu
Boston, MA 02115              Choice is the Choice!
=============================================================

---

From: Hauke Fath <[email-masked]>
Date: Thu, 15 Mar 2001 10:28:19 +0100

>but it seemed not to be trivial.

NFS, designed as stateless (all operations are synchronous), is not really 
fast, itself. And having to emulate file metadata operations that are 
designed into AppleTalk/AppleShare does not make it any faster.

The overall consensus seems to be that supporting the clients' native 
network protocols on the server side (Samba, Netatalk) works out better 
than mapping network protocols on the client side (to NFS, SMB, whatever).

>I believe they all disappeared.  Are any still around that

No. Even the Open Source NFS server project for the Mac was cancelled.

         hauke

--
Hauke Fath                        Tangro Software Components GmbH
                                         D-69115 Heidelberg
[email-masked]                       Ruf +49-6221-13336-0, Fax -21