Subject: Re: CRLF fun stuff again...

From: Duncan Sinclair <[email-masked]>
Date: Fri, 09 Feb 2001 23:38:03 +0000

Hi,

>Someone (sorry, can't remember who) wrote:
>> It is worth noting that the Samba team have point blank refused to add
>> any form of 'translation' to Samba. It serves files exactly 'as is'.

What Samba does is of no great consequence to me.  But I will say
that Windows' CR&LF at least allows files to be manipulated
fairly easy on the Unix side.

Steve Freitas writes:
>I think that is absolutely the right thing to do. I've had this CR/LF thing
>munge up whole ISOs for me, even when it's turned off.

How????

>When you've got to remember to add a line to AppleVolumes.system just to
>keep a file with a new suffix from being corrupted, that's just bad.

Doesn't cr/lf need to be explicitly turned on the in the options
for the share?  But maybe the default AppleVolumes.system wants
to be changed too.  My AppleVolumes.system file has commentary on
this issue that doesn't seem to be in the current version:

 # default translation -- note that CR <-> LF translation is done on all
 # files of type TEXT.  The first line turns off translation for files of
 # unknown type, the second turns this translation on.
 # .       BINA    UNIX
 # .       TEXT    UNIX

The AppleVolumes.system really should be split into two different
files - one with the actual volume info, and a second one for
files types.  I can't remember - can these settings be changed on
a per-share basis?  (Maybe they should even be settable on a
per-directory basis (a la Apache .htaccess file.)

I'll repeat again my position:  if netatalk is going to have this
feature, and there's evidence that people want it, then it should
work properly.  People who don't want this feature should be
unaffected by it.

Cheers,

Duncan.

---

From: Marc Miller <[email-masked]>
Date: Fri, 9 Feb 2001 16:09:04 -0800 (PST)

> >munge up whole ISOs for me, even when it's turned off.

The right sequence of binary code will translate to MacOS linefeed.  If
netatalk intercepts this and translates it to a Unix linefeed, it has just
rewritten a binary file, perhaps even rewritten a program as it's being
copied.  

> per-directory basis (a la Apache .htaccess file.)

AppleVolumes.default has the volume info.  The .AppleVolumes file in the
user's home directory overrides that default if present.

> work properly.  People who don't want this feature should be

I agree.  But if MacOS no longer specifies whether the data is binary or
text when it's being copied in the first place, there's nothing netatalk
can do about it.  If we write a complex algorithm to analyze the file
looking for non-alphanumeric characters in an attempt to determine whether
the file is text or binary, we slow down afpd significantly.  

My suggestion:  talk to Apple.  Get their client side to properly declare
whether the AppleShare transfer is text or binary when it's being copied
and CR/LF will work just fine as is and without damaging binary files.

============================================================
	/\/\arc ._|. /\/\iller ([email-masked])
	Computer Room Consultant
	Information Technology/Lab Management
============================================================
I can be contacted through the Communication Center link from
http://www.mother.com/~mjmiller/

---

From: Donald Lee <[email-masked]>
Date: Sat, 10 Feb 2001 01:07:09 -0600

>work properly.  People who don't want this feature should be

The problem with this feature is three fold.  For one, it is not clear to
me that an algorithm can be devised that can be reliable.  Two, the
conversion is not reversible, and three, the conversion (i.e. corruption)
is absolutely silent.

The first problem is
really much more serious and fundamental than it appears.  That is
that there is really no way to know, reliably, that a file is of type TEXT.

This is because the file type is not intrinsic to the the data.

A good example of this is some applications (ref: archives) that
write out the data *first* and *then* change the type to "text".

The converse also happens.  There are utilities that use TEXT
as their file types that are not TEXT at all.  (most of these, I
admit, are programmer/geek tools)

One _could_ come up with ways to deal with this, but I'm not at all
convinced that they will be "correct".  For instance, do you *really* want
to make netatalk rip through a given file and change all the returns
to linefeeds any time the file type changes to text?  Do you want to
do this even though the conversion is *not* reversible?

If a file type changes from text to something else, do you want to
rip through and change all the linefeeds to returns?

If the cr/lf conversion were symmetric, it would not bother me
so much, but a file that is incorrectly "converted" is utter
trash.  There is no way to take a binary file that has been "converted"
and figure out which of the linefeeds used to be returns.

The third problem is serious when you consider that the potential corruption
of the first two are fatal to your data, and there is no way for netatalk
to know (or report) that you just lost your data.  One could put in
"safety" checks in the conversion code.  For instance, you could check
if a file being converted has any high-bits turned on (a sure sign that it's
not 7-bit ASCII).  OK, what do you *do*???  Would netatalk produce an I/O
error at this point?  Would it simply stop converting?  There is no
clean way to deal with this.

The upshot of all this is that the feature may or may not be useful,
but it is **DANGEROUS** if not fully understood and very carefully used,
and I don't think it is possible to make it safe and reliable.

-dgl-

---

From: Steve Freitas <[email-masked]>
Date: Sat, 10 Feb 2001 02:12:06 -0800

Hi.

> fairly easy on the Unix side.

Duncan, I think the point of mentioning Samba's design is related to the
credibility of their approach. That is, some Very Smart People designed
Samba, and Samba is vastly more widely deployed in production use than
Netatalk. Therefore, there is a high probability that their designs are
correct for the majority of users. This is an issue for all cross-platform
file servers, and their conclusion carries heavy credibility.

>> munge up whole ISOs for me, even when it's turned off.

I detailed this in earlier posts. I'm using 1.5pre3 from Mac OS 8.6. The
short version, if I can remember correctly:

1. I had no CR/LF enabled for the share. Other config files were as default.

2. I took a file on the Mac, duplicated it and gave the files these two
names: "test.iso" and "test.jpg".

3. I checksummed both files on the Mac, then copied them to the Netatalk
volume. I then checksummed them again on the Linux box. Test.jpg was
unchanged, while test.iso had changed.

4. I added a .iso entry to AppleVolumes.system, and that solved it for .iso.

(Disclaimer: This was a while ago and very convoluted, so for the actual
results, check the archives. My mail blew up a few weeks ago and I've lost
all my archives, so I can't refer you to it.)

> this issue that doesn't seem to be in the current version:

I don't know... I discussed it with the list, but everybody I talked to
seemed to think that CR/LF was off, and in fact when I explicitly tested its
functionality, it was off. However, Netatalk was definitely fiddling with my
ISO and corrupting it through linefeeds, and my experimentation showed that.

> work properly.  People who don't want this feature should be

If it's possible for it to work properly, I agree. However, my concern is
that Netatalk's default configuration (1.5prex) currently permits corruption
of binaries, and I think all CR/LF ought to be disabled by default.

Steve

---

From: [email-masked] (Albert Lunde)
Date: Sat, 10 Feb 2001 9:05:57 CST

> of binaries, and I think all CR/LF ought to be disabled by default.

It sounds like there's a strong case for making it a compile-time
option that's off by default. Evict the suspect code.

In these times of the web, the most effective transformation
of end-of-lines will recognize LF CR and CRLF and transform
each into a single end-of-line, but this is definitely NOT
reversible. 

I use a short C program to fix end of lines when I need it,
and use BBEdit for most text eding with the default set to Unix
end of line for new files.

--
    Albert Lunde          [email-masked] (new address)
                          [email-masked] (old address)