Subject: replicated DID error

From: Mike Johnson <[email-masked]>
Date: Tue, 19 Dec 2000 07:47:44 GMT

Hello again everybody!

I'm planning on writing a pretty detailed email so here's the hook: I 
managed to replicated the DID conflict error. I felt it was important to 
first replicate the error before it can be reliably fixed.

(I'm rather unfamiliar with bug reporting and the like, so bare with me. 
I'll just be as detailed as possible and hopefully somebody can discern 
something useful from this. I've got to learn C!)

Anyway, let me first explain my test.

I've been using netatalk for sometime now and I've never experienced 
anything like what many of the people on this list have reported. No DID 
conflicts, dancing icons,and thankfully no disappearing files.

The rpms from the sourceforge site install like a dream and works very 
well the uams.

But the difference between my netatalk use and other's is that I rarely 
use it for production use. I'm pretty much the only one who ever logs 
into that machine, and it's just a temporary file dumping zone and a 
backup server. (I would like to use a similar setup for production but 
the aforementioned problems have kinda scared me away.:)

It seemed pretty obvious to me that whatever problems have arisen, they 
were exclusively multi-user problems, and thus the reason why I haven't 
heard of anybody successfully replicating the problems.

I devised three things that might possibly cause errors. I was looking to 
cause any errors at all.

1st - High network traffic and buffer overruns.
2nd - Removing .AppleDouble files using a Windows machine while a mac was 
copying folders.
3rd - Use a Windows machine to rename folders while a mac is actively 
trying to copy files into them.

After hearing Roland's explanation of DIDs errors and resolving, I 
thought this might be a good first test.

Let me explain the hardware too:

1. Linux Redhat 7.0 Server, Pentium 90 Mhz Vectra. Three IDE hard drives. 
32 Megs of ram. (Hey, it's a test machine, afterall.) I named it 
challenger.... Redhat 2.2.16-22 kernel.
2. My laptop. A Dell Inspiron 3000. lots o ram. SuSE 6.4. Not that this 
matters.
3. Another similar Dell laptop running Windows 2000.
4. A mac. It's a 7200 running mac OS 8.6.

I installed a 100 M/s hub for just these computers. All computers except 
the 7200 were running 100 M/s NICs, the mac 10 m/s.

I took detailed notes through this whole process, but let me just run 
through the juicy parts.

>From the linux dell, I ran "ping -f" to flood the server. I wanted to 
simulate lots of network traffic. It's a ridiculous amount really, but 
very effective.

Before and after this I checked the system log. There were no signs of 
errors or buffer overruns during the whole test. Maybe next time I'll 
ping from more computers.

I ran the ping for a while, then stopped it to check the collisions. 967 
packets collided, 251,648 sent. 3 dropped.

Next, I restarted the ping and mounted challenger with my mac. There was 
no noticeable slowdown.

I started a copy of 400+ megs of assorted data files to challenger -- all 
of them were just random things. Copped home directories, pro/e files, 
IGES, stuff like that.

Things were going well (or should I say badly) for my little test. The 
collisions increased everything the IDE drives kicked in, but this was 
expected.

Just for kicks, I decided to do a find from the windows computer. Using 
samba, I found all of the .AppleDouble files as they were being copied to 
challenger from the mac. I deleted them as they came, but it didn't 
disturb the copy or cause any log entries. I don't really know what 
pertinent information is here that would hinder a copy, but  apparently 
nothing.

Here's the last one I ran:

About halfway through the copy, I went again to the windows computer. 
Using samba, I tried to rename the top level folder that the mac was 
copying. It was called "pocketscience" or something like that. I couldn't 
rename it (file being used error on the windows side), but I could rename 
the two folders directly underneath "pocketscience".

Those two folders were called "viking" and "from_andrews_home". I renamed 
them to "viking2" and "from_andrews_home2". The copy kept moving, 
unfettered by this name change.

I finally stopped the ping to check the stats. Some 20,000 packets 
disappeared, either dropped or collided, with about 2 million sent.

I let the copy go on for a little while; still nothing in the syslog. 
Tired of watching the progress bar, I canceled the copy. I immediately 
realized however, that though I could double-click on "pocketscience" and 
read the contents, the two subdirectories didn't have the new names I had 
given them.

When I double-clicked on each "from_andrews_home" and "viking", the mac 
yelped an error and refused to open the directories.

Then, finally, a syslog entry:

-------------------------
Dec 18 15:16:00 challenger afpd[1765]: WARNING: DID conflict for 'pa' and 
'pa2'. Are these the same file?
Dec 18 15:16:00 challenger afpd[1765]: WARNING: DID conflict for 
'from_andrews_home' and 'from_andrews_home2'. Are these the same file?

<snipping DHCP requests>

Dec 18 15:17:59 challenger afpd[1765]: WARNING: DID conflict for 'pa' and 
'pa2'. Are these the same file?
Dec 18 15:17:59 challenger afpd[1765]: WARNING: DID conflict for 
'from_andrews_home' and 'from_andrews_home2'. Are these the same file?
Dec 18 15:18:47 challenger PAM_pwdb[1765]: (netatalk) session closed for 
user miketec
Dec 18 15:18:47 challenger afpd[1765]: 273819.15KB read, 5.18KB written
Dec 18 15:18:48 challenger afpd[1765]: done
--------------------------

Here I had a choice, either try to fix the problem and salvage the files, 
or try to make them disappear. I choose to try to fix the problem; I'll 
try making them disappear again some other time.

I noticed that simply closing the windows yet still keeping the mac's 
challenger mount didn't refresh the windows. The only thing that 
successfully resolved the DID conflict was disconnecting from the server 
and connecting again.

Restarting the mac didn't make any difference to this test, but umounting 
and remounting was what appeared to matter most.

Thanks for reading this far. The reason I'm sending this to the admin 
list and not to the developers is that I was hoping someone might help me 
out. If somebody has some time and the equipment to reproduce these 
errors following the steps above, it might speed things along. Then I'll 
know for sure whether it was something to do with my equipment, how I 
worked, or if it truly was the error I was looking for.

Thanks,

- Mike Johnson

---

From: "Lawrence Farr" <[email-masked]>
Date: Tue, 19 Dec 2000 08:30:42 -0000

Mike,

I put:

  veto files = /Icon?/ /Temporary Items/ /TheVolumeSettingsFolder/
/.AppleDouble/ /Network Trash Folder/ /.AppleDesktop/
/TheFindByContentFolder/

In my smb.conf. That makes all the Mac invisibles PC invisibles. That cures
one problem
in your test anyway!

     !
   -~*~-
    /!\	Merry Christmas!
   /%;@\
  o/@,%\o	Lawrence Farr
  /%;`@,\	EPC Direct Limited mailto:[email-masked]
 o/@'%',\o	T:01179666123 F:01179666111 M:07970780901
 '^^^N^^^`

> -----Original Message-----
> From: Mike Johnson [mailto:[email-masked]]
> Sent: 19 December 2000 07:48
> To: [email-masked]
> Subject: replicated DID error
>
>
> Hello again everybody!
>

---

From: "Jonathan Newman" <[email-masked]>
Date: Tue, 19 Dec 2000 07:47:40 -0800

> From: Mike Johnson
> Sent: Monday, December 18, 2000 11:48 PM

... snip ...
> unfettered by this name change.

... snip ...

> Then, finally, a syslog entry:

Interesting.  This is one case I hadn't thought of.  The DID database has
stale information in it.  Sounds like we need to do an existence test of the
one found in the database on a DID conflict.

... snip ...

> successfully resolved the DID conflict was disconnecting from the server

This makes sense.  The DID database is stored per client, and the only way
to "destroy" the database is kill the client (disconnect from the server).

Jon

---

From: Mike Johnson <[email-masked]>
Date: Tue, 19 Dec 2000 18:38:55 GMT

> Jon

The DID database on the client.... which client? The client process or on 
the mac itself?

(Thanks -- this is helpful information for the next time I try to break 
things.)

- Mike

---

From: Glenn Trewitt <[email-masked]>
Date: Tue, 19 Dec 2000 17:33:56 -0800

Mike Johnson wrote:
.... many details about testing for DID conflicts ....

Thanks, Mike, for the crucial starting point!

This is with version 1.4b2+asun2.1.4-39-test running on FreeBSD 4.2.  I'm
quite sure that network configuration doesn't matter one little bit.

Here is a much simplified test case that produces the same error.  See the
end for my explanation of what's going on and a brief discussion of what
might be done about it.

Cause a folder named "AAA" to exist on a server (e.g., underneath your home
directory).  Log into the server via AppleShare as yourself.  Log in again
(also as yourself) from another computer.  Do the following:
    Session 1         Session 2        Comments
    ---------         ---------        --------------------
    Open "AAA"        Open "AAA"       Works fine, no surprise.

    Close "AAA"       Close "AAA"

    Rename "AAA" -> "XXX"              Works fine.

                      Close & Reopen   Folder still named "AAA"!
                      home directory   Get the "DID conflict"
                                       log message for "AAA" and "XXX"
                                       from Session 2's afpd process.

                      Open "AAA"       Get Finder error "The folder
                                       "AAA" could not be opened,
                                       because it cannot be found."

Here's some extra credit work:
    Create a new                       Works fine.
    folder named
    "AAA"

                      Close & Reopen   OS 8.6: Now, there are two
                      home directory   "AAA" folders!  Both of them
                                       have the contents of the
                                       newly-created "AAA" folder.
                                       OS 9.0: You see two "AAA"s
                                       for a moment, then the second
                                       one disappears!

I could go on (and I did, but things just got more and more weird).

Explanation:  Each afpd process keeps a map of DID -> directory, all
independent of each other.  As long as directory names don't change, all of
these maps are identical, because (as of this version of netatalk) DIDs are
computed from DEVICE+INODE.

When one afpd changes the name of a directory, it is able to update its DID
-> directory map.  The other afpd process is stuck with the DID pointing at
the old directory name.  Two separate things then happen:
1) When the Mac with the stale afpd process tries to enumerate the contents
of the home directory (when you close and re-open it), it sees a new
directory with the same DID.  When it tries to insert that into its DID ->
directory map, it notices the duplication and logs the error.  (This is from
looking at the code.)  At this point, the Mac hasn't seen a problem.
2)  When you double-click on "AAA" from the stale Mac, the Finder asks afpd
for the folder by its DID and gets the old, now nonexistent, name, returning
an error.

It looks like the code is doing the minimum possible work to deal with this
situation (just enough to keep from getting totally lost).  I'm pretty sure
the code could be improved to detect this situation and handle it silently by
updating the stale afpd's state when it comes across the problem.  However,
this will still cause problems in the Finder (not seeing the new name until
the folder window gets refreshed).  Also (IMPORTANT WARNING) that code may
very well be there from long ago to detect and correct other, much more
pathological, situations.  In that case, changing it could have undesirable
side-effects.  (A comparison of the prior source should be able to answer
this question.)

The correct solution, of course, is to have a persistent, shared database for
the DID->directory mapping.  Then, the afpd clients would stay in sync.
However, there are two problems with this:
    1)  You'll take some sort of performance hit by doing a DB lookup each
          time you enumerate the contents of a directory.  This could be bad.

    2)  If you use symlinks (no loops, please!) to make structured
filesystems for
         sharing folders, a global DID->dirname database would fail, because,

         from the point of view of each afpd process, those directories
really do have
         different names.  (I do this on quite a large scale.)

Thoughts to ponder...

Glenn Trewitt
Lucent Technologies Bell Labs Research
Palo Alto, California

---

From: Basil Hussain <[email-masked]>
Date: Wed, 20 Dec 2000 08:59:10 +0000

Hi all,

> quite sure that network configuration doesn't matter one little bit.

I thought I'd give this a go on the 1.4.99 version
(netatalk-1.4.99-0.20001108mdk installed from RPM to be exact). I was able
to reproduce it almost exactly with that too. This was with a RedHat 6.2
server and two iMacs with MacOS 9.0.4.

> Session 1         Session 2        Comments

Yup. In my error log:

Dec 20 08:43:22 spectrum afpd[12547]: WARNING: DID conflict for 'AAA' and
'XXX'. Are these the same file?

> because it cannot be found."

Got this error too.

> for a moment, then the second

Hmm, well, what you describe for MacOS 9 didn't happen exactly as you say
for me. The two 'AAA' folders just stayed there, just like you say is the
case on 8.6. Neither of them disappeared, even after several minutes.

I hope this information is of use to someone.

Regards,

------------------------------------------------
Basil Hussain ([email-masked])

---

From: Mike Johnson <[email-masked]>
Date: Thu, 21 Dec 2000 00:10:04 GMT

Good sir:

I just tried your same test on an NT server (4.0, sp 6) and had no 
troubles. Immediately after changing the name of a directory on the NT, 
the change appeared on the mac.

Is it possible we could do the same? Any thoughts?

- Mike Johnson

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 12/19/00, 5:33:56 PM, Glenn Trewitt <[email-masked]> 
wrote regarding "DID conflict" log message, an explanation and thoughts:

---

From: Netatalk <[email-masked]>
Date: Sat, 23 Dec 2000 13:44:28 +0100

>     Session 1         Session 2        Comments

Is this with the latest version of netatalk from sourceforge? My clients
see updates as soon as the folder is reread/reopenend. Client 2 should've
seen the update by Client 1, so no AAA dir should appear. The DID conflict
does occur here, too.

>           time you enumerate the contents of a directory.  This could be bad.

Think broader: not only store DID/name info in the Database, but every
information afpd needs to enumerate a directory. This was suggested long time
ago to speed up directory lookups drastically. Then on directory lookup, just
check if the directory changed in between (simple stat). Using shared memory
or mmapped files between the afpd processes with appropriate synchronization,
this could boost directory lookups by factor 3 easily.

But more and more doubts creep up if this is really the 'right' way to do it.
For every solution I find to an update problem (and we'll encounter them) a new
case pops up where this solution produces further problems. I keep thinking
that the easiest and cleanest way would be to add the info stored in the
.Appledouble files into a separate stat-like block in reiserfs and let the
OS take care of these update problems. Great for linux, but what
about *BSD or Solaris? *argl*

Another way: tie this database into the file access functions in libc
(LD_PRELOAD) ... advantage: all file utils (even samba) would automagically
update the DB, too.

>          different names.  (I do this on quite a large scale.)

Why is this the case? The relevant DID imho should be the DID of the target
directory the link points to. Afaik nothing disallows THE SAME directory to
occur in different locations in the file tree. Just store one access path
in the database (preferably the real one without symlinks).

Roland Schulz
[email-masked]