Subject: Re: How to fix CRLF stuff...

From: Duncan Sinclair <[email-masked]>
Date: Tue, 13 Feb 2001 15:29:52 +0000

Hi,

Bob Rogers writes:
>   From: Duncan Sinclair <[email-masked]>
>   Date: Sat, 10 Feb 2001 17:08:46 +0000
>
>   . . .
>
>   As an additional safeguard, it may be wise to be able to mark files
>   which have had the cr/lf transformation done to them so that when
>   they are sent back to the Mac there is no chance of them being
>   messed up.

>   If for some reason the sticky bit was unavailable for use in this
>   fashion, berhaps a spare bit could be found inside the appledouble
>   info - but this wouldn't be as efficient.
>
>Since the original motivation was to make Mac files usable to Unix
>apps/users, using the sticky bit seems like asking for trouble.  If the
>files are only accessed from Macs via shares, then you can guarantee
>that no Unix apps will misinterpret or change their sticky bits.  But in
>the general case, it seems safer to use the .AppleDouble file . . .

OK, I've done some reading up and the sticky bit is out, but maybe
there is some combination of 'x' bits that could be used.  I'm not
100% sure if netatalk uses the exec bits to mean anything (it doesn't
look like it, but I don't fully understand what afpd/unix.c does with
modes), but I was thinking that maybe "g-x,o+x" might be a good flag.

>   1.  If some other Unix app were to edit the file, would it preserve
>"+t"?  (emacs 20.7 does, btw.)  How about some random Windows app (via
>Samba)?

Unix apps should preserve mode bits.  I don't know if samba does this.
>From a quick glance it seems to do something with at least the u+x bit,
but I can't see any particular pattern.

>   2.  Also, if some other bit were used on a different netatalk port
>due to system constraints, then you wouldn't be able to move shares
>between the two systems via Unix means (e.g. dump/restore, tar).

Yep, well life's like that.  With the "g-x,o+x" solution, this should
be portable between all posix compliant systems, unless they are playing
tricks...

>   3.  There is also a compatibility issue here.  After installing a
>hypothetical new version of netatalk that uses this explicit
>transformation indication, there will be transformed files without the
>sticky bit, and possibly untransformed "+t" files.

Yep, an "upgrade" step would be needed, along the lines of what you
suggest.

>This is the same marking operation as when a newly Unix-created file is
>first seen by afpd, and could be as simple as "chmod +t" . . . but
>somehow it feels safer to me to keep it in the .AppleDouble file . . .

Emm...  How about in both places.  With either being used, probably
using the mode bits first for speed.

>   4.  Also, how does this interact with the "noadouble" feature?  It
>seems to me that if "lazy" .AppleDouble creation is enabled, then the
>server would always have to create an .AppleDouble file for text files,
>just to tell itself in the future that the "+t" bit is significant.

Not sure.  I don't use "noadouble", so don't know its implications.
I guess though that "noadouble" would be mostly used in non-Mac areas.

(My motivation is to allow manipulation of Mac files on Unix, not Unix
files on Mac, so I would rarely point netatalk at a "Unix" directory.)

Perhaps "noadouble" should imply "nocrlf", or at least no mode bit
twiddling.  If a "noadouble" area is unix only, it should use the
traditional AppleVolumes.system way of deciding if it wants to do cr/lf
conversion.

Is "noadouble" lazy, or completely prohibitive?

>In addition to automatic transformation, it would be good to have a Unix
>utility, call it afpd-xlate-crlf, that can do the transformation and
>update the appropriate bit on demand.

Good idea.

>That would help for the ambiguous
>and/or misleading cases (as long as the daemon continues to obey the
>bit, regardless of what the file type says).

Hmmm...  If you over-wrote a text file with a binary file, I wouldn't
want it to continue doing cr/lf munging.  But then my helper process
would sort out the move from text to binary as well as binary to text.

>   But, to be safe, such a program should do file locking to avoid
>corruption, since it has to update both the data and the bit.  Which
>makes it seem a bit like your proposed helper app.  Maybe afpd should
>spawn afpd-xlate-crlf to do these translations on the side?

Excellent idea.  You might choose to you the same binary for the helper
app, for the manual process, and for the "upgrade" step...

>The locking
>should give you all the serialization you need (as long as you avoid
>race conditions between afpd and afpd-xlate-crlf).  But if you do
>translate asynchronously, you still have the problem that you can't
>return an error to the client if afpd-xlate-crlf fails.  And maybe,
>after all is said and done, translation isn't such a big job that it's
>worth the overhead of forking a new process . . .

One reason for serialising the operation is to avoid forking several
hundred processes as a result of the Finder copying a large number of
text files and changing their types quickly.

Thanks for your comments!

Cheers,

Duncan.

---

From: Jeremy Buchmann <[email-masked]>
Date: Tue, 13 Feb 2001 09:22:21 -0800

Okay, this seems to be getting very complicated...is there a more simple
approach?

For example, I use the Unix command "less" quite a bit to read through text
files.  But when I type "less picture.jpg", less tells me that it's a binary
file.  How does it know?  I don't know exactly how it works, but I'm
guessing it just scans the first few hundred or thousand bytes and checks
for non-text/non-line-ending characters and if any are there, it reports it
as a binary file.  This may sound overly-simplistic, but it seems to work
incredibly well.  What do you think?  Here are the pros and cons I could
think of:

Good Things:
1) Much more simple to implement
2) Should be pretty effective

Number 1) is the main thing.  Much less work, less complication, fewer bugs.
As for 2), well it should be as effective as less if we used it's method of
examining files (and I've never seen less be wrong).

Bad Things:
1) It's not a perfect way of determining text/binary files.
2) It may be slower.

Regarding 1), sure it's not perfect, but I think it'd be pretty darn good.
I can think of a few things (like tar files with mostly text files and then
a binary file at the end) that might thwart it, but those are special cases.
Besides, can the dual-process-IPC method garantee 100% accuracy?  As for 2),
I don't think it'd be *that* slow (less can do it pretty quickly) and
besides, when have Mac users been concerned with speed?  No one buys a Mac
because it's a speed demon.

Thoughts?

--Jeremy [[email-masked]]

---

From: paul beard <[email-masked]>
Date: Tue, 13 Feb 2001 09:37:09 -0800

MAGIC(5)                                                 MAGIC(5)

NAME
       magic - file command's magic number file

DESCRIPTION
       This manual page documents the format of the magic file as
       used by the file(1) command, version 3.30. The  file  com
       mand  identifies  the  type  of  a file using, among other
       tests, a test for whether the file begins with  a  certain
       magic  number.   The  file /usr/share/magic specifies what
       magic numbers are to be tested for, what message to  print
       if  a  particular  magic  number  is found, and additional
       information to extract from the file.

       Each line of the file specifies a test to be performed.  A
       test  compares the data starting at a particular offset in
       the file with a 1-byte, 2-byte, or 4-byte numeric value or
       a  string.   If  the  test succeeds, a message is printed.
       The line consists of the following fields:

<snip>

---

From: Bob Rogers <[email-masked]>
Date: Wed, 14 Feb 2001 10:43:38 -0500

From: Jeremy Buchmann <[email-masked]>
   Date: Tue, 13 Feb 2001 09:22:21 -0800

   as a binary file.

I don't know about less, but I understand some apps just look for NUL
characters, a dead giveaway for a binary file (at least in the Unix
world).

   This may sound overly-simplistic, but it seems to work incredibly
   well.  What do you think? . . .

Nothing is too simple if it works.  ;-}  But isn't this likely to get
confused by Mac characters and/or internal markup?  If you create some
nontrivial test files (e.g. with font changes, bullet characters) in
SimpleText and save them to a netatalk share, what does `less' do with
them?

					-- Bob

---

From: Matthew Wilson <[email-masked]>
Date: Wed, 14 Feb 2001 08:03:17 -0800 (PST)

> SimpleText and save them to a netatalk share, what does `less' do with

just put some european characters in the file, like a 'u' with the two
dots above it.  Less will warn you that it may be binary.  (try reading a
Pike readme file)

alternatively, you could still have a file that you don't want translated
that didn't happen to have any 'funny' characters in it.

---

From: Jeremy Buchmann <[email-masked]>
Date: Wed, 14 Feb 2001 10:21:09 -0800

(I have three replies quoted here...please note who it is I'm quoting)

> Bob Rogers:

A good idea...or as someone else pointed out, magic is what the 'file'
command uses to figure out what kind of file a file it is.

> Bob Rogers:

Formatting of any kind implies a non-text document.  Those shouldn't be
translated.

The only reason for translation (AFAIK) is to make text documents useable on
the Unix side (my primary interest).

> Matthew Wilson:

[disclaimer: what I know about 8-bit ascii or 2-byte chars couldn't fill a
thimble]

Do these chars have a unique ascii code?  If so, they can be recognized as
text.  If not, then people with those chars in their documents might be SOL.
C'est la vie.

As far as less is concerened, I was just using it as a simple example
showing that some programs can differentiate between text and binary files.

> Matthew Wilson:

That is possible, (like with tar'ed text files?), but either way, we're
playing with the law of averages.  Does Finder always know what is really a
text file and what isn't?  Hell no.  Especially not in a heterogenous
environment.  Hell, half of the files on my desktop have the "blank" icon.
All I'm saying is, as long as our solution won't be perfect, why make it
difficult?

> Peter DiCamillo:
> There is a much simpler approach if the goal is only to avoid the
> "corruption" that happens when applications like the Finder and
> StuffIt change the type to TEXT after creating the file.  While I
> like the idea of comprehensive solution to the CR/LF problem, this
> simpler approach would solve the only serious problem we have with it
> here.  My idea is the following:
> 
> 1. Define a new "suppress CR/LF translation" bit in the AppleDouble
> file.  The bit would be off for files created by previous versions of
> netatalk, and assumed off if the AppleDouble file isn't present.

Ok, I follow so far...

> Peter DiCamillo:
> 2. When afpd creates a new file, the suppress translation bit is set
> if the file is not translated at that time.

Ok...but how does afpd know to translate the file?  This is the problem I'm
talking about...it seems you're talking about the problem of keeping track
of what has or has not been translated.

> Peter DiCamillo:
> With this approach, a Mac user could use the Finder to copy a TEXT
> file to a netatalk volume and back again without the file changing,
> although it would not be translated for Unix users.

This is how it works now unless you specifically set the crlf option in one
of the config files...how has this changed anything?

> Peter DiCamillo:
> This would be 
> compatible with a more comprehensive solution where afpd could be
> told to either translate or not translate a given file.  When the
> suppress bit is off, setting the file's type to TEXT would force
> translation, and would also be useful to the Mac user.

Now I'm confused...how do we set the file's type to TEXT?  Isn't that the
problem...as of OS 8, Finder doesn't set the type before copying?

-- Jeremy  [[email-masked]]