Subject: AppleVolumes.system database

From: Gjermund G Thorsen <[email-masked]>
Date: 19 Jan 2002 14:26:24 +0100

http://ext.comitas.no
-- 
Yours Sincerely,

Gjermund Gusland Thorsen

"I’m just sitting here listening to the scream of the butterfly"

---

From: Charles Mangin <[email-masked]>
Date: Wed, 23 Jan 2002 14:37:11 -0500

i have a client machine on my network running 8.6 (all the other machines are OS 9.x or X) that is having trouble saving files that he has opened over the network. the problem happens in a particularly old and buggy pieces of software which, unfortunately, is crucial to our operations. ain't it always that way?

anyhow, i recently recompiled netatalk 1.5 with "--with-did=cnid" as my previous installation with "--with-did=last" was giving errors all over the place with folders and files disappearing into the void.

those errors are gone now, but the -43 has taken its place.

does anybody know if updating this client machine to OS 9 will help anything, or if there is something else at play that might make things work equally well for everybody?

---

From: Charles Mangin <[email-masked]>
Date: Wed, 13 Feb 2002 09:48:55 -0500

*chirp chirp* quiet around here lately...

i recently rebuilt netatalk on my redhat server to use cnid (and thus, DB3) as i was getting heinous DID conflicts using the default method.

so now, after a few months of otherwise peachy performance, i am seeing this:

Feb 13 09:33:31 localhost afpd[11994]: cnid_lookup: txn_checkpoint: DB_RUNRECOVERY: Fatal error, run database recovery
Feb 13 09:33:31 localhost afpd[11994]: cnid_add: Unable to lookup rootinfo: DB_RUNRECOVERY: Fatal error, run database recovery

there's no performance or other weirdness from the client side (or if there is, they're not telling me...)

a: what is going on here? i don't know dick diddly about DB3

b: how do i fix it? it says to run recovery, but i can't seem to find a command to run that will actually run any kind of recovery....

any help would be greatly appreciated.

---

From: Matthew Keller <[email-masked]>
Date: 13 Feb 2002 10:00:27 -0500

Don't use CNID unless you understand the ramifications of enabling an
"experimental" service. It's getting better, but still has issues.

-- 

Matthew Keller
Enterprise System Analyst
Computing & Technology Services
Information Services Division
State University of NY at Potsdam
Potsdam, NY USA

http://mattwork.potsdam.edu/

---

From: Matthew Keller <[email-masked]>
Date: 13 Feb 2002 10:02:40 -0500

Wow, on re-reading my response I realized it was rather rude- Public
and humble apologies for that. Very much not my intention. *slaps self*
	CNID is currently not stable, and is not recommended for use by
non-developers/testers and certainly not in a production environment.
It'll be real cool when it's done but in the mean-time, it's kinda icky
for general use.

-- 

Matthew Keller

---

From: Thomas Kaiser <[email-masked]>
Date: Wed, 13 Feb 2002 16:12:12 +0100

On Wed, 13 Feb 2002 09:48:55 -0500, Charles Mangin wrote:

> as i was getting heinous DID conflicts using the default method.

What do you refer to as 'default' (last or hash aka mtab) and which Netatalk
version are you running?

I never ever got any messages regarding DID conflicts when using the 'last'
DID scheme, so I'm a bit curious.

Regards,

Thomas

---

From: Charles Mangin <[email-masked]>
Date: Wed, 13 Feb 2002 11:44:42 -0500

>DID scheme, so I'm a bit curious.

i guess it was "last". i had such bad problems with directories spontaneously disappearing, error -50s, unmovable items, etc, that i had to resort to trying a different scheme. for one reason or another, mtab wasn't working or it didn't compile or something (i forget why. maybe i didn't have mtab set up properly? that doesn't sound right...) anyhow, i decided to try cnid, as it looked like a good idea at the time.

like i said, it worked beautifully for a couple of months. and, so far, i haven't noticed any errors on the user side, just the messages in the log. with DID=last, the logs filled up with did conflicts, and the users lost a lot of stuff.

---

From: Charles Mangin <[email-masked]>
Date: Wed, 13 Feb 2002 11:50:25 -0500

>It'll be real cool when it's done but in the mean-time, it's kinda icky

apology accepted, i guess. you still need to send me something pretty to make up for it, though :)

anyhow, under the default scheme (i guess it's "last") i had awful problems with directories spontaneously disappearing, error -50s, unmovable items, etc. so much so that i had to resort to trying a different scheme. i decided to try cnid, as it seemed like a good idea at the time.

like i said, it worked beautifully for a couple of months. and, so far, i haven't noticed any errors on the user side, just the messages in the log. with DID=last, the logs filled up with did conflicts, and the users lost a lot of stuff. however, i wouldn't exactly call my system a "production environment". the server is only used for transferring files from one machine to another, and not any real, permanent storage. it's just faster to have a central fileserver than to have a bunch of peers filesharing. and if i have to rebuild the server once in a while, hell, it's job security :)

---

From: Charles Mangin <[email-masked]>
Date: Wed, 13 Feb 2002 11:52:54 -0500

>"experimental" service. It's getting better, but still has issues.

well, since i wrote, i found the db3 db_recover utility, but it doesn't do a lot unless i run it in the same directory as the db to recover. i'm still stumbling about my filesystem trying to find it, but any help or pointers would be appreciated (or a filename to look for...)

---

From: Robert Thiemann <[email-masked]>
Date: Wed, 13 Feb 2002 14:12:51 -0500

hey All,

It's the resident netatalk newbie here.

I'm coming from an NT Server and moving to Linux. Our shares were set 
up a little differently on the NT side. For example, we have a share 
called "Faculty" that looks like this:

Faculty
|->jdoe
|->sdavis
|->rmaster

And what I had before was (the glory of NT's ACLs) the Faculty 
folders set with permissions so that the individual faculty member 
had owner privileges, none of the other faculty had access to another 
faculty's folder, and the BackUp admin had full rights to the root 
Faculty share, and every subsequent folder.

Under Linux, i've got it set up so that when Faculty log in, they get 
their home directories exported as a share from a structure that 
looks like this:

/home
    /shares
       /faculty
          drwx------   jdoe faculty      4096 Feb 13 10:53 jdoe
          drwx------   sdavis faculty      4096 Feb 13 10:53 sdavis
          drwx------   rmaster faculty      4096 Feb 13 10:53 rmaster

So I need to back this thing up, and since I'm using Retrospect ( the 
only thing I've found that safely stores the Mac file info, resource 
fork, etc.)

I'm thinking what I could do is create a .AppleVolumes file for root, 
and export out a raw share for the root of the Faculty directory, 
which will then let retrospect go in and grab whatever files need to 
be grabbed without regard to permissions.

Is there any major drawback to maintaining a login via AppleShare as 
root that way? (the Backup server is separate from our file server).

This isn't the only directory I'll need to do this one. There's about 
four others too. And it'll make life a lot easier for me since 
Retrospect will grab additions to the folders within that share 
without me having to constantly having to modify the backup sets.

So, other than having a user with way too much power logged in, is 
there any reason I shouldn't do this?

Anyone have a better solution?

-- 
Robert Thiemann
Network Engineer
VCU Adcenter
[email-masked]

---

From: Thomas Kaiser <[email-masked]>
Date: Wed, 13 Feb 2002 22:32:27 +0100

On Wed, 13 Feb 2002 11:44:42 -0500, Charles Mangin wrote:

[ID scheme causing trouble]
> i guess it was "last". i had such bad problems with directories spontaneously
> disappearing, error -50s, unmovable items, etc,

Hmm... This sounds like the problems with the older inode-kludge code, the
predecessor of mtab/hash.

You reported problems a while back and wrote, that you were using an early
1.5pre. Someone mentioned on the netatalk-developer list, that until 1.5pre6
(can't remember exactly) mtab has been the default and from then on, last.

I'm using last for years and had never had any strange occurances of these
symptoms, you reported.

> that i had to resort to trying a different scheme.

I would give 'last' a second chance on a production server, as the only
well-known problems with it are non-working file-/folder-IDs. But I've never
heard of data loss or problems with duplicate IDs and so on...

> [...] maybe i didn't have mtab set up properly?

In the doc folder exists a README.mtab document for this purpose...

> [...]
> with DID=last, the logs filled up with did conflicts, and the users lost a lot
> of stuff.

Still can't believe it. Do others had this problems with DID=last?

Regards,

Thomas

---

From: [email-masked]
Date: Thu, 14 Feb 2002 11:31:53 -0500 (EST)

WARNING! LONG POST! (Sorry... -sth)

Robert,

tar (the *NIX tape archiver, which will write to files as well as tapes)
does a fine job of grabbing resource forks, so long as you tar the entire
share directory (which includes the .AppleDouble directory, where the
resource forks live). My backups like on a Linux server, so tar makes
sense, but it could be adapted for use in your environment, if Retrospect
is your backup method of choice.

I'll propose a mechanism in a sec, but first address "why not root doing
backups from the Retrospect server?":

1) Your Retrospect server would probably need to have a clear-text copy of
your Netatalk machine's 'root' password somewhere on its filesystem in
order to do the login. This means that you effectively have that password
written down someplace, which increases its chances of
exposure/compromise.

2) The password would need to be sent to the Netatalk server over the
network, and while the channel would be encrypted (so long as you're using
a fairly recent AppleTalk client), a hijacked server on the Unix end could
be made to cough up that password to interlopers (see any number of
arguments in favor of asymmetric-key-based authentication -- there's a
good explanation on www.openssh.org, I think...).

[Warning: neither I nor anyone I represent assumes any responsibility for
any loss or damage that occures resulting directly or indirectly from the
proposals contained in this email message. Unless you contract me to do
work. Then it's a different story. ;-)]

How about this:

First, make a user called 'retrospect' with some awful password that
you can use in a retrieval script, and no login shell; under RedHat, this
would be done as such:

	[root@benzene root]# useradd retrospect -s /bin/false

Then, there are two options:

1) Duelling cron Jobs
[Caveat: this creates a race condition. You are timesync-ing your clocks,
right?]

a) Netatalk machine runs a cron in the middle of the night that tars up
your target directories, places them in ~retrospect/, chowns them to
retrospect:retrospect, and chmods them to 0600, such that you wind up with
something like this:

/home/retrospect:
-rw-------    1 retrospe      retrospe   34110867 Feb 13 03:15 backups.tar

b) soon after, once you're sure the backup will be done on the Netatalk
machine, Retrospect machine runs its timed job, logs into Netatalk machine 
as 'retrospect', snags that file, and deletes it.

The drawbacks: 1) there's that nasty race condition -- if for any reason
the Netatalk machine isn't finishing tarring the target directories when
Retrospect comes looking for its backup file, you'll get an incomplete
backup on your Retrospect machine, and it might delete the archive out
from under the ongoing tar process; and 2) all your users' backups are
sitting, for a time, in the home directory of another user who can log in
via AppleTalk.

To get around the race condition, see option #2, below. As far as the
backups sitting on the filesystem with "sub-root" ownership well, if
you're going to be moving them with AppleTalk, I can't think of another
way.

2) Scripted, tightly-restricted SSH control of the process

I won't detail this here, since it's a fairly (*ahem*) information-rich
topic (read: difficult to explain with any brevity), but you could script
an SSH client on the Retrospect machine (running MacOS? AppleScript should
be able to handle this) to use "single-purpose keys" -- keys that can only
do one thing once logged in -- to login as 'root' to the Netatalk machine
and coordinate the process. Advantages? Decently secure, since only root
is able to work with the backup archive; no race condition, since one
machine can run the steps in sequence. Drawbacks? Somewhat complex setup,
plus now you've got to guard your Retrospect machine jealously as it now
contains keys that can be used to log into the Netatalk machine as 'root'
and work with backups (but not do anything else). If you're already
superparanoid about your Retrospect machine, though, this probably isn't a
stretch.

Write to me off-list if you'd like specifics about this process. I'm
working on a package ("Sauropod") that is already running this sort of
thing: a Linux machine with a tape drive runs it and can back up other
*NIX machines via SSH/2 or NFS, and Novell fileservers. Probably SMB
(Windows) and AppleTalk soon, too. (I'm working it up with a plug-in
acrhitecture.)

Good luck, and best regards to all the folks who make this list a great
resource for a great product.

-sth

--
Sam Hooker
Computer Audio/Internet Services
hbc, LLC
(802)846-1845
www.hbconcoctions.com

---

From: [email-masked]
Date: Thu, 14 Feb 2002 12:35:20 -0500 (EST)

> resource forks live). My backups like on a Linux server, so tar makes

Stupid fingers: "My backups liVe on a Linux server..."

*drat*

-sth

---

From: [email-masked] (Paul Sander)
Date: Thu, 14 Feb 2002 23:55:44 -0800

Rather than log in as "root" on the Netatalk machine, why not log in as
an ordinary user, and grant super-user access to specific tasks for which
it's absolutely necessary?  There are a number tools that provide this
capability, "sudo" being one of the more popular.  I use "op" with much
success.

--- Forwarded mail from [email-masked]

--- End of forwarded message from [email-masked]

---

From: Simon Hobson <[email-masked]>
Date: Fri, 15 Feb 2002 10:30:51 +0000

[email-masked] wrote :

>and coordinate the process.

The problem with both of these ideas is that Retrospect does NOT 'log in'
to the server using anything other than AFP. I don't think it even supports
'non-standard' UAMs (I certainly found that it won't use the Microsoft UAM
to access an NT server).

Ie it :

* Makes an AFP connection to the server (over IP or AppleTalk as appropriate).

* Copies whatever files are required to be copied according to the rules of
the script it is executing, and the access priviledges it has.

* Closes the connection.

As far as I know, there is NO method whereby you can insert some extra
layer in between.

>as 'retrospect', snags that file, and deletes it.

Whilst this would work, it would completely defeat the object of using
Retrospect in the first place - which is to back up the Mac files AS MAC
FILES. Not only that, instead of having to restore both the file(s) and
it's .AppleDouble partner(s) (as would be the case using tar or cpio on the
Unix/Linux side, you have to restore the backup file for the whole server
before using tar or cpio to extract the required files. It also means that
Retrospect can't do incremental backups.

I think the best option would be to persuade Dantz to make a client for
Linux that understands the file structure of a netatalk share. But you
still have the dilemma of whether to run it as root.

The only other method I can think of is to use AppleScript to mount the
volume when the script is ready to run (Retrospect has the ability to call
an AppleScript handler at certain points). This would allow you some more
flexibility in how you make the connection - but you still have to mount
the netatalk share using AFP.

Simon

-
Simon Hobson, Technical Services Engineer
Colony Gift Corporation Limited
Lindal in Furness, Ulverston, Cumbria, LA12 0LD
Tel 01229 461100, Fax 01229 461101

Registered in England No. 1499611
Regd. Office : 100 New Bridge Street, London, EC4V 6JA.

---

From: Charles Mangin <[email-masked]>
Date: Fri, 15 Feb 2002 08:53:56 -0500

so, i'm apparently taking somewhat of a risk in running netatalk 1.5.0 with DID scheme=cnid, and thus the errors i'm getting are probably my own damned fault.

woohoo.

anyway, is there a preferred version of the berkley db3 software i should be running for the cnid scheme? my system already has 3.1.17 on it (redhat 7.1ish) though i see that there are RPMs for 3.2.9-4 and a tarball for 4.0.14

testing the RPM, i see that it breaks a lot of dependencies (and thus, i could spend another day doing the RPM dependency dance..) and i'm wary of installing the tarball for similar reasons - a lot of stuff apparently depends on the version of the db3 libs i have installed currently

---

From: Robert Thiemann <[email-masked]>
Date: Fri, 15 Feb 2002 09:52:16 -0500

>As far as I know, there is NO method whereby you can insert some extra

My line of reasoning exactly. Plus the .AppleDouble file will become 
a known irritant when i have to go in and grad just _one_ file off of 
a back up tape. Retrospect handles all of it for me.

Since I manually log in to the server from the Retrospect machine 
anyway, basically, what I want to know is, is there any _Real_ 
problem with me leaving shares on my Retrospect box mounted as root? 
(Please read the previous line carefully) Aside from someone breaking 
into my server closet, and stealing files that way (which, if they've 
broken in, odds are they're going to get what they want regardless), 
does it:
A) provide any other potential security risks that I need to be aware 
of? I will _not_ be setting to server to keep the password. The 
shares are mounted at boot, and it prompts for the password.

b) Is there a better way?

It's my understanding there's no way to make another user who has the 
ability to ignore file permissions, but not actually run anything, so 
I can't create that. And since I can't log into the same server from 
the Retrospect server under different accounts to the same server 
(not without making one hell of an Apple Script). I'm really wanting 
to back this thing up as close to the "mac way" as possible. I'm 
aware of the wonders of tar, and amanda, but I'm also aware of the 
nightmares caused by back up programs that don't realize that each 
individual mac file has it's own resource fork, and not some resource 
fork collective called .AppleDouble.

-- 
Robert Thiemann
Network Engineer
VCU Adcenter
[email-masked]

---

From: Simon Hobson <[email-masked]>
Date: Fri, 15 Feb 2002 15:42:36 +0000

Robert Thiemann wrote:

>anyway

Another thought, has anyone done any experiments to see what happens to
permissions if you have to restore files from Retrospect ? (says he
thinking "it's about time I looked at this myself" !)

Simon

-

---

From: [email-masked] (Paul Sander)
Date: Fri, 15 Feb 2002 10:29:47 -0800

I was under the impression that the idea was to use Retrospect to create
backups via FTP to some server, then log in to the server to initiate
a backup of the Retrospect archives to tape.  If that is indeed the case,
then using Secure Shell for the second half is very doable and utilities
such as sudo and op suddenly have value.

This overall process can then be initiated and controlled from the client
machines.

>--- Forwarded mail from [email-masked]

>--- End of forwarded message from [email-masked]

---

From: Robert Thiemann <[email-masked]>
Date: Fri, 15 Feb 2002 15:10:07 -0500

>such as sudo and op suddenly have value.

No, the idea was to back up the AppleTalk shares via Retrospect (on a 
separate machine, obviously) to tape.

The server I need to back up is my production server. We have a 10 
tape autoloader that is used to back up via Retropsect. Im migrating 
from NT4, where I just mounted the shares, used NT's ACLs to give the 
Backup Operate full control of the shares being backed up.

We're an almost complete Mac-shop except for the server and two PCs. 
I need to make sure the Resource info doesn't get munged.

-- 
Robert Thiemann
Network Engineer
VCU Adcenter
[email-masked]

---

From: Steve Freitas <[email-masked]>
Date: Fri, 15 Feb 2002 12:12:59 -0800

You must be running 3.3.11, no exceptions. Get it here:

http://www.sleepycat.com/update/3.3.11/db-3.3.11.tar.gz

Just ./configure, make, make install. That should put it in /usr/local, I
believe. Then add the following line to /etc/ld.so.conf:

/usr/local/BerkeleyDB.3.3/lib

Then run:

[yadda@blah]$ ldconfig

If you want to verify that the library loader has found the db3 libs, do:

[yadda@blah]$ ldconfig -p | grep Berkeley

You should see three of the libraries listed in the proper directory.

Now you're ready to recompile Netatalk with it pointed to the new DB3
installation.

Steve

> From: Charles Mangin <[email-masked]>
> Date: Fri, 15 Feb 2002 08:53:56 -0500
> To: <[email-masked]>
> Subject: DB3
> Resent-From: [email-masked]
> Resent-Date: Fri, 15 Feb 2002 08:55:21 -0500 (EST)
> 
> so, i'm apparently taking somewhat of a risk in running netatalk 1.5.0 with
> DID scheme=cnid, and thus the errors i'm getting are probably my own damned
> fault.
> 
> woohoo.
> 
> anyway, is there a preferred version of the berkley db3 software i should be
> running for the cnid scheme? my system already has 3.1.17 on it (redhat
> 7.1ish) though i see that there are RPMs for 3.2.9-4 and a tarball for 4.0.14
> 
> testing the RPM, i see that it breaks a lot of dependencies (and thus, i could
> spend another day doing the RPM dependency dance..) and i'm wary of installing
> the tarball for similar reasons - a lot of stuff apparently depends on the
> version of the db3 libs i have installed currently
> 
>

---

From: Thomas Schierle <[email-masked]>
Date: Sat, 16 Feb 2002 12:18:13 +0100

> I need to make sure the Resource info doesn't get munged.

I don't remember your original request, but may be Netatalks admin
group feature is what you are looking for? -- A member of admin group
is granted supervisor rights for all sharepoints provided by Netatalk.

-Thomas

-- 
Thomas Schierle, Munich, Germany

PGP key [DSS/DH] 0xA23CDA1D available at various public key servers

---

From: Robert Thiemann <[email-masked]>
Date: Mon, 18 Feb 2002 09:15:58 -0500

>is granted supervisor rights for all sharepoints provided by Netatalk.

Hmmm... where's the documentation for this live?

-- 
Robert Thiemann
Network Engineer
VCU Adcenter
[email-masked]

---

From: Thomas Kaiser <[email-masked]>
Date: Mon, 18 Feb 2002 15:24:49 +0100

> Hmmm... where's the documentation for this live?

There is a couple of files in the doc directory of a netatalk source tree.
CONFIGURE is the one, you are searching for. 'man afpd.conf' will also be
helpful.

Regards,

Thomas

---

From: Charles Mangin <[email-masked]>
Date: Mon, 18 Feb 2002 09:48:20 -0500

>believe. Then add the following line to /etc/ld.so.conf:

thanks.

just to nitpick, to build the 3.3.11 distro, i needed to cd to build_unix then ../dist/configure

it's compiling on my machine right now, and i'll let you know how the rest goes...

---

From: Charles Mangin <[email-masked]>
Date: Tue, 19 Feb 2002 14:27:41 -0500

well, with the new(ish) 3.3.11 version of db3 compiled and installed, and netatalk 1.5.2, all my DID and db_* errors seem to have gone away. the logs are still getting a little cluttered, tho, as i may have enabled debugging messages when i rebuilt netatalk, and every time someone connects, i get messages like:

"Feb 19 13:47:40 localhost afpd[12928]: CNID DB initialized using Sleepycat Software: Berkeley DB 3.3.11: (July 12, 2001)"

but it's better than filling up with DID conflicts, eh?