Subject: How to fix CRLF stuff...

From: Duncan Sinclair <[email-masked]>
Date: Sat, 10 Feb 2001 17:08:46 +0000

Hi,

Now that we've (hopefully) established that netatalk does do a
reversable transformation on text files as they are written and read
and doesn't corrupt files by design, I'd like to put forward my current
thoughts on how the actual bug could be fixed.

The problem is that text files created with a non-"TEXT" type and then
changed to have a "TEXT" type don't have the cr/lf transformation done
to them.  This means then when they are served, because they are now
recognised as "TEXT", they will have an un-necessary cr/lf
transformation done to them.

Either the un-necessary transformation should be avoided, or else when
the type of a file is changed, a transformation should be done, if
needed.

I previously wrote:
 Here's my proposed solution:
  * A second "helper" process would run along side the main afpd process,
    tied to it with pipes.
  * tasks would be handed to it like "do a cr/lf transformation of this file".
  * it would have to take care to lock files.
  * tasks are done serially - to make sure nothing gets screwed.
  * there may be other types of thing it could be used for.

As an additional safeguard, it may be wise to be able to mark files
which have had the cr/lf transformation done to them so that when
they are sent back to the Mac there is no chance of them being
messed up.  In fact, this could be done without my proposed changes
above.  Peter DiCamillo made a similar suggestion previously, and I
thank him for bringing it to my attention last week.

I would mark transformed files by setting one of the lesser-used mode
bits on the file.  (This is similar to how the "XBITHACK" feature
works in Apache, and it may not surprise you to learn that I was the
inventor of this idea.  (I didn't do the implementation however.))

So, what would happen is that when a transformation is done (either at
the time of writing the file, or by my "helper" process as a result
of a change of file type), one bit of the file's mode, say the sticky
bit, would be set on.  If for any reason the file was changed to a
different type and so un-transformed, the sticky bit would be cleared.

When netatalk comes to serve the file, if it is a Mac file (complete
with AppleDouble info), then it would examine the sticky bit.  If it is
set it would do the transformation as it serves the file.  This would
ensure that it only re-transforms files which have already been
transformed.

On the unix side there is a clear indication that the file has had this
transformation - a nice bonus.  Unix-created files would continue to
use the file extension to determine whether the file is text and want
transformed when served.

If for some reason the sticky bit was unavailable for use in this
fashion, berhaps a spare bit could be found inside the appledouble
info - but this wouldn't be as efficient.

Constructive criticism welcome.  Offers of code even more so.

The more I deleop this idea, the closer it gets to being implemented.
The stuff about setting the mode bit could be done in an afternoon,
perhaps, if you knew your way around the code.  The stuff about the
helper process would require greater effort, but could be worth it
in the opportunities for extra features it would give.

Thanks for your time.

Cheers,

Duncan.

---

From: Bob Rogers <[email-masked]>
Date: Mon, 12 Feb 2001 12:42:07 -0500

From: Duncan Sinclair <[email-masked]>
   Date: Sat, 10 Feb 2001 17:08:46 +0000

   . . .

   thank him for bringing it to my attention last week . . .

I think the basic idea -- an explicit indicator -- is terrific.  It also
has the advantage of being robust with respect to AppleVolumes.system
changes, since afpd makes the decision once, and remembers what it
decided.

Since the original motivation was to make Mac files usable to Unix
apps/users, using the sticky bit seems like asking for trouble.  If the
files are only accessed from Macs via shares, then you can guarantee
that no Unix apps will misinterpret or change their sticky bits.  But in
the general case, it seems safer to use the .AppleDouble file . . .

   Some points in that vein:

   1.  If some other Unix app were to edit the file, would it preserve
"+t"?  (emacs 20.7 does, btw.)  How about some random Windows app (via
Samba)?

   2.  Also, if some other bit were used on a different netatalk port
due to system constraints, then you wouldn't be able to move shares
between the two systems via Unix means (e.g. dump/restore, tar).

   3.  There is also a compatibility issue here.  After installing a
hypothetical new version of netatalk that uses this explicit
transformation indication, there will be transformed files without the
sticky bit, and possibly untransformed "+t" files.  (Does Samba use +t?)
So, as a postinstallation step, it seems to me that you would need to do
the following:

	if CR<->LF transformation is enabled,
	    for each .AppleDouble file,
		if a text file,
		    mark as transformed (assuming it already had been)

This is the same marking operation as when a newly Unix-created file is
first seen by afpd, and could be as simple as "chmod +t" . . . but
somehow it feels safer to me to keep it in the .AppleDouble file . . .

   4.  Also, how does this interact with the "noadouble" feature?  It
seems to me that if "lazy" .AppleDouble creation is enabled, then the
server would always have to create an .AppleDouble file for text files,
just to tell itself in the future that the "+t" bit is significant.
[Caveat:  I couldn't find the documentation for "noadouble", after a
casual search, so I may have misremembered something.]

In addition to automatic transformation, it would be good to have a Unix
utility, call it afpd-xlate-crlf, that can do the transformation and
update the appropriate bit on demand.  That would help for the ambiguous
and/or misleading cases (as long as the daemon continues to obey the
bit, regardless of what the file type says).

   But, to be safe, such a program should do file locking to avoid
corruption, since it has to update both the data and the bit.  Which
makes it seem a bit like your proposed helper app.  Maybe afpd should
spawn afpd-xlate-crlf to do these translations on the side?  The locking
should give you all the serialization you need (as long as you avoid
race conditions between afpd and afpd-xlate-crlf).  But if you do
translate asynchronously, you still have the problem that you can't
return an error to the client if afpd-xlate-crlf fails.  And maybe,
after all is said and done, translation isn't such a big job that it's
worth the overhead of forking a new process . . .

   My $0.02 (or probably less, seeing as I have no code . . . ;-)

					-- Bob Rogers