Subject: Re: CRLF fun stuff again...

From: Duncan Sinclair <[email-masked]>
Date: Sat, 10 Feb 2001 10:18:52 +0000

Hi Folks,

I've received a number of emails about all this stuff.  This is the
only one I think really _needs_ a reply...

Donald Lee writes:
>At 5:38 PM -0600 2/9/01, Duncan Sinclair wrote:
>>I'll repeat again my position:  if netatalk is going to have this
>>feature, and there's evidence that people want it, then it should
>>work properly.  People who don't want this feature should be
>>unaffected by it.
>
>The problem with this feature is three fold.  For one, it is not clear to
>me that an algorithm can be devised that can be reliable.  Two, the
>conversion is not reversible, and three, the conversion (i.e. corruption)
>is absolutely silent.

Let's address these points then...

>The first problem is
>really much more serious and fundamental than it appears.  That is
>that there is really no way to know, reliably, that a file is of type TEXT.

You know a file is text by two indicators:
  1) Its file type is TEXT.
  2) It has an extension which tells netatalk it is text.

These two indicators cover files created by Macs and files created
by Unix, repectively.

With a Mac-created file you can change the extension as much as you
want and it will not change how netatalk interprets the file.  Only
if it is marked as "TEXT" will the cr/lf transformation be used.

With a unix created file, netatalk doesn't know if it is text or not,
so it uses the extension to decide whether to do a transform or not.
If a unix file is edited on a Mac it becomes a Mac file at this point.

>This is because the file type is not intrinsic to the the data.

True.

>A good example of this is some applications (ref: archives) that
>write out the data *first* and *then* change the type to "text".

My suggested enhancement to netatalk would catch this and do the conversion
on the fly.

>The converse also happens.  There are utilities that use TEXT
>as their file types that are not TEXT at all.  (most of these, I
>admit, are programmer/geek tools)

This should not matter.  The transformation is reversable and so
is transparent to the Mac application.

>One _could_ come up with ways to deal with this, but I'm not at all
>convinced that they will be "correct".  For instance, do you *really* want
>to make netatalk rip through a given file and change all the returns
>to linefeeds any time the file type changes to text?

Yes.  And all the linefeeds to returns.

>Do you want to
>do this even though the conversion is *not* reversible?

But the transformation is reversible.  the conversion is as follows:

  CR -> LF
  LF -> CR

You do the transformation twice and you get back an identical file.

>If a file type changes from text to something else, do you want to
>rip through and change all the linefeeds to returns?

Yes.  Change all the linefeeds to returns, and all the returns to
linefeeds.

>If the cr/lf conversion were symmetric, it would not bother me
>so much, but a file that is incorrectly "converted" is utter
>trash.  There is no way to take a binary file that has been "converted"
>and figure out which of the linefeeds used to be returns.

It's a transformation, not a conversion and it is symmetric.

>The third problem is serious when you consider that the potential corruption
>of the first two are fatal to your data, and there is no way for netatalk
>to know (or report) that you just lost your data.  One could put in
>"safety" checks in the conversion code.  For instance, you could check
>if a file being converted has any high-bits turned on (a sure sign that it's
>not 7-bit ASCII).  OK, what do you *do*???  Would netatalk produce an I/O
>error at this point?  Would it simply stop converting?  There is no
>clean way to deal with this.

If it's handled right you never get corrupted files.  You don't (and,
yes, shouldn't want to) go checking to see if the file "looks" like
text.

>The upshot of all this is that the feature may or may not be useful,
>but it is **DANGEROUS** if not fully understood and very carefully used,
>and I don't think it is possible to make it safe and reliable.

Well I think it can be fixed to make it safe and reliable.

Cheers,

Duncan.

---

From: Steve Freitas <[email-masked]>
Date: Sat, 10 Feb 2001 02:27:02 -0800

> You do the transformation twice and you get back an identical file.

Not true with all files. Some CR-based files have some LFs in there for
other purposes. When your conversion turns all CRs into LFs, you've wiped
out the ability to turn the LFs back to CRs without disturbing the original
LFs. All LFs become CRs, and your data is corrupted. That's why this
transformation qualifies as high-potential corruption when applied to
anything but a text file. And we cannot always be sure what is text and what
is not, which means this is unsolvable.

Steve

---

From: Sak Wathanasin <[email-masked]>
Date: Sat, 10 Feb 2001 11:03:19 +0000

In reply to Duncan Sinclair's message of the 10/02/2001 at 10:18 +0000,

>...

No it isn't, not if the file was a binary file in the first place. 
Think of a binary pixmap embedded in a PS file. There will some bytes 
in the pixmap that are \0x0A that should not be converted to CRs.
-- 
Sak Wathanasin
Network Analysis Limited
178 Wainbody Ave South, Coventry CV3 6BX, UK

Internet: [email-masked]
Phone: (+44) 24 76 41 99 96        Mobile: (+44) 79 70 75 19 12 
Fax: (+44) 24 76 69 06 90

---

From: "Marc J. Miller" <[email-masked]>
Date: Sat, 10 Feb 2001 19:43:36 -0800

>These two indicators cover files created by Macs and files created

Okay, I agree with the former, but the latter is a little more 
complicated.  String compares are slow.  For each file copied, the name has 
to be checked against the following list (and this is just off the top of 
my head)

.txt
.htm
.html
.c
.pl
.ksh
.sh
.cc
.p
.bat
.vbs
.asc
.ans
.cgi
.troff
.am
.cfg

(you get the idea)  There are also file extensions like .sys which are 
sometimes text (config.sys and msdos.sys, for example) and sometimes binary 
(cdrom.sys, ansi.sys, and msdos.sys in versions of DOS before Windows 
95).  File extensions are a guide to the user for guessing the file 
type.  I've observed that most Unix and Mac people don't use file 
extensions at all.  The scripts in /etc/rc.d/init.d on Red Hat systems 
don't have extensions, yet they're all text.  Mac folks seem to be quite 
used to relying on the icon of the file.  Ever tried to save an HTML file 
in Netscape and then edit it in SimpleText?  Many versions of SimpleText 
won't open it at all because Mac is so dependent on the resource 
information to tell it what programs should be able to open it.  File 
extensions were introduced through DOS, really.  Unix and MacOS see the "." 
in the file name as just a character.

It may be possible to work with some library that the "file" command uses 
for identifying file types (if it uses a library), but as I'm not 
interested in activating the CRLF feature in the first place, I'll let 
someone else look that up.  :)

---

From: Bob Rogers <[email-masked]>
Date: Mon, 12 Feb 2001 11:18:03 -0500

From: "Marc J. Miller" <[email-masked]>
   Date: Sat, 10 Feb 2001 19:43:36 -0800

   Okay, I agree with the former, but the latter is a little more
   complicated.  String compares are slow.  For each file copied, the
   name has to be checked against the following list (and this is just
   off the top of my head) . . .

If necessary, this table lookup could be hashed for speed (if it isn't
already).  But I would be very surprised to learn that the speed
difference was even measurable, compared to the other overhead of
opening & serving a file.  String compares aren't *that* slow.  And
since this lookup only applies for files created by Unix (or Samba),
it's even less of an issue.

   . . . I've observed that most Unix and Mac people don't use file
   extensions at all.

I beg to differ; who else but a Unix person would think of compound
extensions like .tar.gz, or .i386.rpm ?  And could you imagine `make'
without extensions?

   The scripts in /etc/rc.d/init.d on Red Hat systems don't have
   extensions, yet they're all text.

They are scripts, actually, which conventionally have no extension.
(Except for the odd ".sh" or ".csh" script, but then you have to type
the extension at the shell prompt, which is probably why using an
explicit extension isn't popular.)

   . . . File extensions were introduced through DOS, really . . .

;-}  Next thing you know, you'll be suggesting that Bill Gates invented
GUIs and the Internet . . .

   Until the early 80's, IIRC, file systems almost universally required
extensions; Unix was the (then) rare exception in that it didn't parse
"." specially.  Then the Mac (actually, the Apple Lisa) changed
everything . . .

   Thanks in advance for being a good sport,

					-- Bob Rogers

---

From: Bill Moran <[email-masked]>
Date: Mon, 12 Feb 2001 11:46:30 -0500

> GUIs and the Internet . . .

I thought it was Al Gore who did that?

---

From: Duncan Sinclair <[email-masked]>
Date: Mon, 12 Feb 2001 18:17:00 +0000

>it's even less of an issue.

Plus the fact that the Mac (via File Exchange) does a similar extension
scan on file types it doesn't recognise...  I haven't heard anyone
complain that File Exchange slowed down their Mac or was a waste of
time.

Cheers,

Duncan.

---

From: Marc Miller <[email-masked]>
Date: Mon, 12 Feb 2001 14:00:44 -0800 (PST)

> extensions like .tar.gz, or .i386.rpm ?  And could you imagine `make'

Alright... correction:  tar and gz do require that extension, but most of
the time, shell scripts aren't marked, nor are binary executables.  File
extensions are more of a convention than a requirement.  You could run
make without file extensions at all as long as the Makefile specifies the
file names.

> GUIs and the Internet . . .

No, he bought out the company who had the original concept for
Windows.  :)  What I mean is that until DOS, file extensions were just a
naming convention.  You want to designate that a file is a text
file?  Then put it in a directory called "text" or "txt."  Naming
conventions were there, but they didn't gain the massive popularity they
have now until DOS came along and started requiring .exe, .com, and .bat
for executables.  But we're getting off-topic here.

> "." specially.  Then the Mac (actually, the Apple Lisa) changed

Ah.  Well, then I stand corrected.  

============================================================
	/\/\arc ._|. /\/\iller ([email-masked])
	Computer Room Consultant
	Information Technology/Lab Management
============================================================
I can be contacted through the Communication Center link from
http://www.mother.com/~mjmiller/

---

From: "Chris Garrigues" <[email-masked]>
Date: Mon, 12 Feb 2001 17:03:51 -0600

> From:  Marc Miller <[email-masked]>
> Date:  Mon, 12 Feb 2001 14:00:44 -0800 (PST)
>

I'd like to see you try...Make would be next to useless w/o extensions.

> for executables.  But we're getting off-topic here.

That's BS.  The use of extensions (and a lot of other things) 
in MSDOS was based on the same in CP/M and CP/M got most of those same things 
from RT-11 (or was it RS/X?).

Most of the various competing minicomputer OSes also used extensions in a 
similar way.

> Ah.  Well, then I stand corrected.  

If your knowledge of computer history only goes back two decades, you 
shouldn't be making these kind of pronouncements.

Chris

-- 
Chris Garrigues                 http://www.DeepEddy.Com/~cwg/
virCIO                          http://www.virCIO.Com
4314 Avenue C                   
Austin, TX  78751-3709		+1 512 374 0500

  My email address is an experiment in SPAM elimination.  For an
  explanation of what we're doing, see http://www.DeepEddy.Com/tms.html 

    Nobody ever got fired for buying Microsoft,
      but they could get fired for relying on Microsoft.

---

From: "John R. Dunning" <[email-masked]>
Date: 13 Feb 2001 05:07:23 -0000

[Getting to be off-topic; sorry...]

    From: "Chris Garrigues" <[email-masked]>
    Date: Mon, 12 Feb 2001 17:03:51 -0600
[...]    
			      Make would be next to useless w/o extensions.

Correct.  It's extremely rare any more to find a makefile which
doesn't rely on pattern rules, eg rules driven by semantic knowledge
of filename extensions.
[...]
		The use of extensions (and a lot of other things) 
    in MSDOS was based on the same in CP/M and CP/M got most of those same things 
    from RT-11 (or was it RS/X?).
    
RT had variants with no filesystem, as well as extremely simple
filesystems (not much more than a disk driver), so it's somewhat
arguable whether extensions were a feature there.  RS/X (often
pronounced R-sucks :-) made heavy use of filename extensions as far
back as the early '70s or so.

Right.  TOPS-10/20 in particular was an early player in that game;
that one dates back (I think) to the late 60s; you can argue that it's
not a competitor, though the groups at DEC would likely have
disagreed.  I remember hacking on other late-60s early-70s systems by
long-forgotten outfits like Interdata and Harris Microsystems; they
all had filename extensions that meant something.

*nix was notable in its early days for being the maverick, and *not*
having any kind of convention for carrying file type information out
of band from the data.  Since it (later) completely overwhelmed the
industry, it's currently fashionable to forget that there were earlier
systems which did a much more comprehensive and consistent job of
dealing with the issue.

I've been doing this *way* too long :-{