Subject: Re: Aliases: Are alternate filesystems worth trying?

From: "Richard Goldman" <[email-masked]>
Date: Thu, 7 Jun 2001 01:18:31 -0400

Well, gotta retract a few things here, sorry folks:

If an XFS volume is created as I've suggested (in this case on a 36GB raid 5 md), the "df -i" command reports that I have 4,480,192 inodes to play with -- certainly less than the 16,777,216 that can be encoded in an alias.

However, after stress-testing and adding about 100,000 new files and directories to the raid, I have inode numbers higher than 16,777,216, which is not safe.  Matthew Geier, I believe, had mentioned that XFS inodes may "encode a disk address..." and not be members of a fixed pool.

So, off to some other filesystem.  Any comments on Reiser or ext2 with regards to putting a limit on total inodes (and of course where and inode number won't exceed in value, the total number of inodes)?

By the way... here's how you can see the inode encoding within an alias:

Using Apple's resedit, open the alias resource and look at the six hex digits beginning with position 73.  These six digits, when converted to binary (4 bits apiece) and appended to each other, yield the 24 bit binary equivalent of the decimal value of the inode (such as from the "ls -i filename" or "stat filename" commands).

It also appears, at least as of the CVS from the morning of June 6 (PDT), the new mtab stuff isn't having an impact on the encoding.

-- Richard

---------- Original Message ----------------------------------
From: "Richard Goldman" <[email-masked]>
Reply-To: <[email-masked]>
Date: Wed,  6 Jun 2001 14:36:26 -0400

>Thank you all so much for the explanations and leads on understanding the issue of aliases/DID's/files/etc.
>
>A few notes so far:
>
>Linux Kernel-2.4.5-XFS appears to be a stable platform for netatalk.  Download the full source tree from SGI's CVS (this will ensure that you're getting the 2.4.5 kernel).  Know that this pre-patched XFS-cabable kernel distrib from SGI is missing two key files from linux/drivers/scsi/aic7xxx (Adaptec driver).  These files are aic7xxx_reg.h and aic7xxx_seq.h.  Find these files from the kernel.org distrib of the stock 2.4.5 -- and only this version, as the Adaptec driver underwent a major patch between the 2..4.4 and 2.4.5 kernels.
>
>Remember to compile the utilities as well, and lower your alias/DID problems by lowering your inode count on XFS by creating the filesystem with this command:  mkfs.xfs -i -size=2048 /dev/whatever.  The the -size subdirective is really asking for a ratio of inodes to to fs blocks.  With Linux XFS frozen at 4k blocks, by asking for 2048 bytes here, your telling the system to be made with roughly 2 inodes per block instead of the built-in default of -size=256 giving you 16 inodes per 4k block.  That would quickly bust the 16,777,216 inode limit addressable in the 24 inode-dedicated bits in the current alias mapping scheme (which you get when you don't invoke --enable-lastdid in ./configure).
>
>I noticed in the CVS this morning that Jeff had added the new configure option on --with-did={last,mtab).  The mtab option points to an implementation of Bob Rogers scheme to devote more of the alias address space to inodes (raised to 28 bits?  29 bits? from the current 24).
>
>Does anyone know if the implementation is complete, i.e. that the new afpd.mtab file is being read, and this new dev-inode scheme is active and usable if --with-did=mtab is specified?
>
>Regards,
>Richard
>
>

---

From: Mac Conin <[email-masked]>
Date: Thu, 07 Jun 2001 09:34:41 +0200

We' re running 1.56 on suse 7.1 (kernel 2.4) and a raid system of ~200GB
df -i shows

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             102592   32103   70489   32% /
/dev/sda3            4294967295       0 4294967295    0% /data1
/dev/sdb1            4294967295       0 4294967295    0% /data2

and it works, (with little backdraws  resulting from a presumed misconf.
by mayself)

-- 
--schnipp-- 

mit freundlichen grüssen

-----------------------------------------------------------
Mac Conin                        EUC Online Service GmbH
Geschäftsführer                  Taubengasse 9
[email-masked]                    D 50676 Koeln
http://www.euc.de                HRB Köln 32038
tel : +49-221-923 27 33          fax : +49-221-239 651
-----------------------------------------------------------
EUC sponsort Clean up Cologne - eine Iniative der IHK Köln,
der Abfallwirtschaftsbetriebe Köln (AWB) und dem
Grünflächenamt der Stadt Köln. http://www.clean-up-cologne.de

---

From: jeff <[email-masked]>
Date: Thu, 07 Jun 2001 09:31:18 -0400

Richard Goldman wrote:
-- SNIP -- 
> It also appears, at least as of the CVS from the
> morning of June 6 (PDT), the new mtab stuff isn't
> having an impact on the encoding.

Are you compiling with --with-did=mtab? It only affects one line in
etc/afpd/enumerate.c, if I'm not mistaken.

Thanks,
Jeff

---

From: "Andrew D. Myers" <[email-masked]>
Date: Thu, 7 Jun 2001 15:26:33 -0400

Executive summary:

Does anyone know of a handy tool/technique for taking the resource fork
and attributes from an AppleDouble file and stitching 'em into the
corresponding NTFS data streams under Win2K Services for Macintosh?

More detailed situation:

I've got approximately 140GB of data on a samba/netatalk server
(netatalk 1.3.x, don't remember which revision, on linux 2.2.x), in tens
of thousands of files.  I need to move 'em to a Win2K box w/Services for
Macintosh installed.

Here's the complicating factors:  This server's been used by Mac, Linux,
and Windows folks.  The Windows guys have oodles of filenames with >31
characters (which don't appear for the Mac folks), the Linux folks have,
in some cases, written out files which differ only by case (eg: foo.txt
and Foo.txt), and the Mac people 'round here love to use
windows-verboten characters like ? and : in their filenames.

I need to a) copy all the files with data/resource forks & file types
intact, and b) make all files visible to the Mac guys on the new server,
and c) use Win2K legal filenames (duh).

Here are the viable solutions I've come up with:

1.  Write a perl script to crank through the exported shares on the
netatalk server which renames the files that will cause Windows fits
(ie, replace invalid chars with ~), and mangles names that are > 31
chars or differ only in case (foo vs. FOO).  Then, once all the
filenames are in order, go do a drag'n'drop from one server to the other
on a Mac, and go home for the weekend.

2.  Write a perl script to do similar as #1, but instead of renaming,
actually copy the files from the netatalk server to the Win2K box.

Problem with #1:  I'm extremely cagey about turning a script loose on
140GB of valuable production data, even if it's only renaming stuff.  If
at all possible, I'd like to leave the old server absolutely intact so
we can fall back if (when) we run into problems during the migration.

Problem with #2:  I can't find an easy way to take the AppleDouble
information and put it into the appropriate NTFS file streams, short of
learning me the AppleDouble file format and writing a tool to do this
m'self.  I've searched high and low for such a beastie, but have yet to
find one.

At this stage, it looks like I'm going to go with option #1 (after a
suitable backup).

Anyhow, apologies for the long-winded explanation (thus the summary up
front), and thanks for any suggestions.

--Andrew

---

From: "Brandon Caudle" <[email-masked]>
Date: Thu, 7 Jun 2001 17:20:43 -0400

I suggest samba for this you can move stuff between any unix distro(or
linux) to windows. Its file mangling char is ~ you can email me if you have
any questions.

Brandon Caudle

---

From: "Brandon Caudle" <[email-masked]>
Date: Thu, 7 Jun 2001 17:29:24 -0400

Forgot to state something

if you used samba all you would have to do is set up samba (easy) and go to
the windows 2000 server go to the share \\server\share and drag and drop the
files.  on the share the mangling char is = ~ and it would do it for you.  I
did this but recently went to all linux machines@ home but we still have
some at work

Brandon Caudle

---

From: Matthew Geier <[email-masked]>
Date: Sat, 09 Jun 2001 09:26:52 +1000

> However, after stress-testing and adding about 100,000 new files and directories to the raid, I have inode numbers higher than 16,777,216, which is not safe.  Matthew Geier, I believe, had mentione…

 While I wouldn't say for sure with out reading the XFS 'whitepapers', I
understand from the XFS list, that Inodes are not fixed in number. XFS
allocates more inode space as required.
 All df -i will do is tell you what is currently allocated (I think). If
you ever ran out of inodes the file system would allocate more.

 The inode numbers on XFS are 64bit and encode the address of either the
inode block or the first extent of the file.

 Even with the new mtab scheme, with 64 bit Inodes on XFS , there is
going to be a massive loss of information when 34 bits of the inode
number are thrown away.
 And you can't configure an XFS file system with a smaller than 2^30
inodes to prevent the clash from happening.

 Presumably its impossible to have more than 2^30 files on a HFS disk
over the entire lifetime of the filesystem. Seems like a bit of a
serious long term restriction?.

---

From: Jonathan Paisley <[email-masked]>
Date: Mon, 11 Jun 2001 13:27:50 +0100 (BST)

> corresponding NTFS data streams under Win2K Services for Macintosh?

I saw some replies to this which talked about using samba alone. Clearly
this won't work because of the macintosh data.

To extract the mac info from the netatalk .appledouble files, you can link
with netatalk's libadouble library. Then there are functions to read/write
either fork and also extract mac flags and type/creator info.

As far as the streams on the W2K server go, I think they're something like
this:

	:Afp_AfpInfo
	:AfpResource

The AfpResource stream is simply the raw data from the resource fork. The
:Afp_AfpInfo is a small file (about 60 bytes IIRC) containing mac file
info. I haven't figured out the exact format of this file, but I should
imagine that it shouldn't be too hard to synthesize it.

You can read/write these streams using smbclient or smbfs.

The problems you mentioned with funny characters are trickier. SFM uses
reserved unicode code positions for representing illegal windows
characters that are used in macintosh file names (e.g. : and \, as well
as trailing spaces). In order to successfully transfer over these files,
you'll need to talk to the W2K server in unicode (or come up with some
other name mangling of your own in plain ASCII). I don't think samba
supports unicode directly at the moment.

Does this help any?

-- 
Jonathan Paisley
[email-masked]