Subject: Re: CRLF fun stuff again...

From: Duncan Sinclair <[email-masked]>
Date: Sat, 10 Feb 2001 11:11:40 +0000

Hi,

Sorry for the continuing mailbox pollution...

Steve Freitas writes:
>I wrote:
>> But the transformation is reversible.  the conversion is as follows:
>> 
>> CR -> LF
>> LF -> CR
>> 
>> You do the transformation twice and you get back an identical file.
>
>Not true with all files.

Yes true for all files.  No matter what the 8-bit values "13" and "10"
mean, you can do this swap twice and you'll get back the exact same
file.

Here's a test with a binary jpeg file....

  (Using GNU tr - Sun's "tr" doesn't cope with binary files - if you
  repeat this test make sure you use one that does.)

quartz:~% sum duncan.jpg
32290    17 duncan.jpg

Original file's checksum is 32290

quartz:~% tr '\r\n' '\n\r' < duncan.jpg > foo.jpg
quartz:~% sum foo.jpg
13378    17 foo.jpg

Transform once, checksum is 13378

quartz:~% tr '\r\n' '\n\r' < foo.jpg > bar.jpg
quartz:~% sum bar.jpg
32290    17 bar.jpg

Transform a second time, checksum is back to 32290

Convinced yet???

Cheers,

Duncan.

---

From: Cliff Crawford <[email-masked]>
Date: Sat, 10 Feb 2001 08:35:44 -0500

> mean, you can do this swap twice and you'll get back the exact same

That's NOT true.  Suppose you had a file with the following data in it:

43 0A A2 7B 0D DD 38 0A 8E 0D 43 2F
   ^^                ^^

There are two LFs already in the file (I underlined them above).  Now do
CR->LF conversion:

43 0A A2 7B 0A DD 38 0A 8E 0A 43 2F

And now try to convert back:

43 0D A2 7B 0D DD 38 0D 8E 0D 43 2F
   ^^                ^^

The two LFs that were originally in the file have been converted into
CRs.  There is NO way to tell that they were originally LFs.  The file
is now irreparably corrupted.

> Transform a second time, checksum is back to 32290

No.  You must have got lucky and used a file which didn't have 0x0A in
it.

-- 
Cliff Crawford               http://www.people.cornell.edu/pages/cjc26/
                             print "Just another Python hacker"

---

From: Paul Schinder <[email-masked]>
Date: Sat, 10 Feb 2001 09:38:03 -0500

>mean, you can do this swap twice and you'll get back the exact same

No, it isn't reversible.  If a file originally contains both CR and 
LF (most binary files), and you convert the CR to LF, it's no longer 
possible to tell which LF's are converted CR's and which LF's were 
there to begin with.  If you do the reverse transformation, then 
there will be no LF's in the file where originally there were some. 
You simply fooled yourself with a single binary file that happened to 
have only CR.  Try one that has both to begin with, and you'll find 
it's corrupted at the end.

You're also wrong about the Mac OS side.  Not all files that are text 
have file type TEXT, and not all files that are file type TEXT are 
text files.  TEXT is simply a label attached to a file by a program. 
The operating system makes no effort to ensure that TEXT files are 
actually text files.

Turning on CRLF conversion is a very bad idea.  ftp has the same 
problem, and the same data corruption occurs if you're not careful.

-- 
--
Paul J. Schinder
NASA Goddard Space Flight Center
Code 693
[email-masked]