Subject: Disappearing files.....

From: "lists-mail" <[email-masked]>
Date: Thu, 30 Nov 2000 00:49:22 -0000

I not using Quark.

I'm currently pulling down the 2000-netatalk-admins.mail to search through. I 
have been seeing a strange problem for the last two months and trying to pin 
it down.

Periodicly I will hear a complaint from a developer that they have lost 
permissions on a mounted directory. I have them unmount the volume and remount 
it, the problem disappears. Within a few hours, files-entire directories will 
disappear. As in gone, without a trace, vamoosed.

I'm currently pulling down the syslog and messages files from the server to 
search for clues.

Has anyone else been experiencing this? Today we lost several hundred files 
and an entire days code. I gotta fix this, I've started each developer to a 
local disk backup policy until I can solve the problem.

Pertinent data follows,

Thanks, DAve

=======================

All files are BBedit text files, PHP/CSS/JS/HTML/SQL
12 Mac G4s running OS 9.02-9.04
Slackware 7.0 file server
netatalk-1.4b2+asun2.1.3 as a package
uname -a => Linux dev 2.2.13 #3 Mon Aug 21 14:31:39 PDT 2000 i686 unknown

afpd.conf:
"DEV" -address 192.168.3.11 -nodebug -noguest -savepassword 
"ARES" -address 192.168.3.11 -nodebug -noguest -savepassword 

atalkd.conf:
eth0 -phase 2 -net 0-65534 -addr 65280.120 

AppleVolumes.default:
/usr/local/apache/htdocs DEV access=@rbldev
/usr/local/apache/htdocs/ARES ARES access=@li

---

From: Frank Joerdens <[email-masked]>
Date: Thu, 30 Nov 2000 09:52:25 +0100

> disappear. As in gone, without a trace, vamoosed.

Did you fsck the volumes, as in e2fsck -f -v /dev/[your disk] after
unmounting them, and were they without errors? Is this SCSI or IDE? I
just had a maybe related issue with a couple new externeal 50 G SCSI
drives I stuck into my netatalk/samba fileserver which up until then had
only had IDE devices in it, one of which was a CDRW IDE device for which
you have to use SCSI-IDE emulation under Linux. This did apparently not
work together with the Tekram SCSI controller driver that I had
installed. I had corrupted filesystems and vanishing data after about a
couple of days of smooth operation with the new drives until I removed
the CDRW from the computer and the SCSI-IDE stuff from the kernel. Now
it's been working fine for about 3 weeks.

Hope it helps,

Frank

---

From: Chip Mefford <[email-masked]>
Date: Thu, 30 Nov 2000 08:17:07 -0500

Hi Dave;

Thankfully, I have not had this happen.

I have heard reports about this over the 
last few years, but never read anything amounting
to the finding of an actual bug. 

What I *have* had happen, was one of
my users from time to time will page me
and claim they cannot empty their trash
because they don't have permission. 
  When they open their trash can, there
  will be hundreds of files in there, that
  are not their files, but rather someone
  elses. I have *no idea* how this occurs. 

At that point, I'll go into their
netword trash folder on the server and
copy the files back out to the shares based
on the owner, and then delete the offending
trash folder, and all is well. 

I have also read about others experiencing
this as well. 

Never read any resolutions. Or even read where
anyone can replicate this sort of behavior. 

Something is a bit strange I think. 

lists-mail wrote:
> snip

---

From: "lists-mail" <[email-masked]>
Date: Thu, 30 Nov 2000 17:52:47 -0000

> it's been working fine for about 3 weeks.
Well we did a restart this morning, arrgggggggg! Linucks doesn't do a fsck at 
each restart. I'll have to wait until the end of the day and umount the drives 
and check them.. The drive is a Quantum Fireball 13gb IDE drive. The corrupted 
fs is a excellent thought, I'll certainly check it out.

The log shows a very disturbing item, The lines below were just prior to the 
'incident' of the directories disappearing.

Nov 28 16:42:23 dev afpd[19886]: iconopen: open /usr/local/apache/htdocs/
.AppleDesktop/8/8BIM.icon: No such file or directory
Nov 28 16:42:23 dev afpd[19886]: afp_addicon: No such file or directory

*****  9872 lines of the previous errors removed. Over one per second!

Nov 29 13:48:00 dev afpd[14455]: atp_rresp: Connection timed out
Nov 29 13:48:10 dev afpd[14455]: afp_die: asp_shutdown: Connection timed out
Nov 29 14:05:57 dev afpd[21772]: afp_alarm: child timed out
Nov 29 14:35:29 dev afpd[21932]: dsi_stream_read(-1): Connection reset by peer
Nov 29 14:35:29 dev afpd[21932]: dsi_stream_write: Broken pipe
Nov 29 14:35:29 dev afpd[21932]: dsi_stream_write: Broken pipe
Nov 29 15:01:11 dev afpd[19886]: afp_openfork: ad_open: Permission denied
Nov 29 15:01:36 dev last message repeated 7 times
Nov 29 15:02:30 dev last message repeated 4 times
Nov 29 15:03:27 dev last message repeated 4 times
Nov 29 15:03:56 dev last message repeated 2 times
Nov 29 15:19:07 dev afpd[19886]: afp_alarm: child timed out
Nov 29 15:58:12 dev afpd[21960]: afp_openfork: ad_open: Permission denied
Nov 29 15:58:17 dev afpd[21960]: afp_openfork: ad_open: Permission denied
Nov 29 16:19:10 dev afpd[22174]: afp_openfork: ad_open: Permission denied
Nov 29 16:25:25 dev afpd[20344]: afp_alarm: child timed out
Nov 29 16:43:57 dev afpd[22174]: iconopen: open /usr/local/apache/htdocs/
.AppleDesktop/J/JVWR.icon: No such file or directory
Nov 29 16:43:57 dev afpd[22174]: afp_addicon: No such file or directory
Nov 29 16:43:57 dev afpd[22174]: iconopen: open /usr/local/apache/htdocs/
.AppleDesktop/J/JVWR.icon: No such file or directory
Nov 29 16:43:57 dev afpd[22174]: afp_addicon: No such file or directory
Nov 29 16:43:57 dev afpd[22174]: iconopen: open /usr/local/apache/htdocs/
.AppleDesktop/J/JVWR.icon: No such file or directory
Nov 29 16:43:57 dev afpd[22174]: afp_addicon: No such file or directory

The files started going bye bye sometime between 2:00 pm and 3:pm, by 3:30 
people noticed directories had disappeared. very odd. Has anyone else seen log 
entries like these before?

DAve

--

---

From: "lists-mail" <[email-masked]>
Date: Thu, 30 Nov 2000 18:12:39 -0000

OK the afpd errors were appearing because the .AppleDesktop directories were 
missing. I did make a copy of he one remaining and distribute it through the 
volumne, only beacuse I cannot figure out how to create a new one!

Filebuddy doesn't seem to create or update the .AppleDesktop on netatalk 
drives, umounting the drive and remounting it also doesn't seem to create a 
new .AppleDesktop directory.

Does anyone know what _does_ recreate or update these? (nope, don't read C, 
but I'll look anyway, never know ;^)

Thanks,

DAve

---

From: Netatalk <[email-masked]>
Date: Mon, 4 Dec 2000 17:58:46 +0100

> local disk backup policy until I can solve the problem.

Hi,

My 10c on the disappearing files problems ...

1st: the Mac does store some information from a mounted volume. When I unmount
and mount a volume under Mac, on mount the Mac clearly requests some items
from this folder that are not fetched from the volume. Some of my patches
to netatalk make this clearly visible:

Dec  4 09:46:14 server2 afpd[10506]: ASIP session:548(2) from x.x.x.x:2051(0)
Dec  4 09:46:14 server2 afpd[10506]: login xx (uid x, gid x) 
Dec  4 09:46:51 server2 afpd[10506]: afpd: did de44->16d33  
Dec  4 09:46:52 server2 afpd[10506]: afpd: did 16d33->2ca73  
Dec  4 09:46:52 server2 afpd[10506]: afpd: did 2ca73->2  
Dec  4 09:46:52 server2 afpd[10506]: afpd: did 29ad9->16d33  
Dec  4 09:46:53 server2 afpd[10506]: afpd: did 26b34->16d33  

This is an example of a mount from a Mac. The volume is mounted at startup.
Now the Mac requests a directory id (DID) of 0xde44. If the afpd visited
the directory with this DID before, line 3 should never occur, so the Mac
really stores information about the volume on its own. The lines show that
this DID is resolved correctly down to root-level (DID 2), as are two other
directory lookups, both not in afpds directory cache.

That this directory is resolved correctly is due to my patches to afpd.
On a normal base afpd, the lookup would've failed, forcing the Mac to do a
full lookup on the complete directory path, part by part. So no problem here?

Maybe not ... as afpd (at least the versions I looked at) is known to
reuse the DIDs it gives out between sessions, this scenario might happen:
- afpd hands out DID 456 which the Mac stores ... the Mac then unmounts.
- another session removes directory corresponding to DID 456 and creates a new
  directory somewhere completely different which just happens to be assigned
  the same DID by afpd.
- Mac 1 now mounts again, looks around some directories, chances upon # 456,
  caches its DID, and then, when the Mac does a lookup on DID 456, finds the
  new directory _without the Mac noticing_ it got the wrong one.
- what happens now if the user of this Mac now deletes that directory ?

This might account for some of the file loss problems some people seem to
experience. The window between 1 and 2 and between 2 and 3 could be
arbitrarily large. The Mac seems to store DIDs for a long time, even
when their resolving fails.

Solution: never reuse DIDs

CU

Roland Schulz
[email-masked]

---

From: "lists-mail" <[email-masked]>
Date: Mon, 4 Dec 2000 18:04:49 -0000

> [email-masked]

Seems like a good enough explanation to me. That also expains  one of my 
problems as I can see that very scenario unfolding just prior to the last 
incident. Ok, I vow to never reuse DIDs again.......   and how is that done?

;^)

DAve.

--

---

From: "lists-mail" <[email-masked]>
Date: Mon, 4 Dec 2000 18:10:56 -0000

Frank Joerdens <[email-masked]> said:

> unmounting them, and were they without errors? 
<snip>
> Frank
Yes I came in this weekend and umounted the volumes and fsck'd. Everything was 
without error. Currently a reboot seems to have removed the problem for the 
time being. I have had _some_ volume mounted continuously for the last 90 days. 
I currently have the users umounting each night. And I may have Netatalk 
restated once a week, I'll see how this goes.

Thanks,

DAve

--

---

From: Mike Johnson <[email-masked]>
Date: Mon, 04 Dec 2000 17:46:38 GMT

A most excellent explanation!

This problem has been the one thing that has kept me from really using 
netatalk, and I'm sure that I'm not the only one. Especially in a samba 
and netatalk environment, it doesn't take much imagination to think of 
how many directories could be changed / moved / deleted without afpd 
knowing about it.

If there is any way to patch this / fix this I'm sure many, many people 
would be extremely grateful!

- Mike

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 12/4/00, 8:58:46 AM, Netatalk <[email-masked]> wrote

---

From: BWS - Offwhite <[email-masked]>
Date: Mon, 4 Dec 2000 13:17:47 -0600 (CST)

I believe you can compile Samba to be Netatalk aware.  It will then move
and delete the .AppleDouble files properly, or so I have read.

Brennan Stehling - web developer and sys admin
projects: www.greasydaemon.com | www.onmilwaukee.com | www.sncalumni.com

Do you ever find it is easier to explain an idea if you put it into
Star Trek terms?

---

From: "lists-mail" <[email-masked]>
Date: Mon, 4 Dec 2000 21:53:53 -0000

I have seen this mentioned before as a possiblity, but never by experienced 
users. Might I have to be the first to try?

DAve

---

From: andrew morgan <[email-masked]>
Date: Mon, 4 Dec 2000 14:46:22 -0800 (PST)

This has been discussed recently on the samba-technical mailing list.
Yes, there is a --with-netatalk configure option, and yes there are
functions to handle netatalk files, but no, it does not do anything.  The
functions are there, but they are not tied into the samba internals.

No one on samba-technical jumped up and said they would fix this right
away, but maybe someone on this list has the inclination...

	Andy

---

From: "lists-mail" <[email-masked]>
Date: Mon, 4 Dec 2000 23:39:09 -0000

> away, but maybe someone on this list has the inclination...

One more reason to accept a co-workers offer to learn C.

Currently I'm getting heavy use as usual but netatalk seems to be behaving 
itself. It may be as simple as resetting the permissions at all levels, 
deleting all dot directories, then rebooting.

Though, I did follow a suggestion I found in the archive;

(from the root of my appleshare volume)
#mkdir .AppleDesktop
#cd .AppleDesktop
#mkdir 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z  
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

I then spent some time to massage the AppleVolumes.system to ensure the files 
we routinely use were mentioned (director-.dcr .cct, flash, and bbedit were 
not).

Only then did I allow Macs to mount volumes. Once done, this has been the 
first time I have never had any errors whatsoever concerning Icon files in my 
syslog. Funny that these never seem to _always_ be created when needed. Though 
in all honesty I have never paid them the attention I should have, until now.

Thanks,

DAve

---

From: Russell Kerrison <[email-masked]>
Date: Tue, 05 Dec 2000 11:05:47 +1100

> Solution: never reuse DIDs

I wrote a week or two ago about a fault where a great slab of folders had
vanished from the server, and have since discovered that the odd OS9 users
periodically finds entire root folders in their Trash. Thankfully they
notice before they empty their trash!

I could imagine that this might be caused by an incorrectly recycled DID?

Russell.

---

From: "Lawrence Farr" <[email-masked]>
Date: Tue, 5 Dec 2000 11:38:43 -0000

Is there a server side fix in the pipeline for this?

Relying on a user to do the server's job for it doesn't
fill me with confidence in netatalk. It currently junks
about 5Gb a week on average (There's 750Gb on a couple of
servers). 

Luckily I have a fast tape drive.

Lawrence Farr
EPC Direct Limited mailto:[email-masked]
T:01179666123 F:01179666111 M:07970780901

---

From: Chip Mefford <[email-masked]>
Date: Tue, 05 Dec 2000 07:32:26 -0500

'Morning List;

The Disappearing files thread moves on. 

last week, I had one of my users loose a directory
of files.

It kinda worked like this:

/netatalk share
 |
 |--/Source Directory
 |    |
 |     /dir1
 |    |
 |     /dir2 and so on
 |
 |--/Destination Directory

Anyway, under this share, he moved via
drag and drop on a MacOS 9.0.4 all the
contents of "Source Directory" into
"Destination Directory" and all went
well right up until he got a MacBoop
and a dialogue box that stated:

Folder suchansuch could not be moved
to "Destination Directory" because
"Destination Directory" can not be
found.

And as you might expect, all
the files from suchansuch went
missing off into space, never to
seen again.

I had them on last nights backups
so I restored them, and off we went.

All of the rest of the move went just
fine, no problems. It was in the neighborhood
of a dozen directories containing about 
300 files averaging in size of about 
20 megs each or so. No big deal.

But the files just went away. 

Where did they go?

---

From: Chip Mefford <[email-masked]>
Date: Tue, 05 Dec 2000 10:58:00 -0500

What's a DID?

---

From: "Jonathan Newman" <[email-masked]>
Date: Wed, 6 Dec 2000 07:15:54 -0800

> disappear. As in gone, without a trace, vamoosed.

FWIW, after running netatalk in various versions for well over a year
on RedHat boxen, the only behavior I have seen similar to this came
after Adrian's introduction of the "persistent DID kludge" in build
39.  The evidence of the kludge being the problem was the presence of
"DID conflict for ..." in the message log.  I believe that this
problem could extend to deleting one directory could delete the
"wrong" directory.  Depending on the device numbers and number of
inodes on your volumes, the problem may or may not evidence itself.
The bottom line is that the kludge remaps the device and inode numbers
into the DID.  Unfortunately there just aren't enough encoding bits
available to encode both and maintain uniqueness in a 32 bit quantity.
There is a flag in the sourceforge's release  ./configure script to
turn the DID kludge off (--enable-lastdid).

Other than that, we are serving 40+ GB with a Linux box to 30+ people
(none of them running Quark ;) ) without major problems.

Jon