Subject: Re: [netatalk-admins] Problems with sherlock searching netatalk

From: "Paul Khristolubov" <[email-masked]>
Date: Wed, 17 Mar 1999 09:03:56 +0700

Hello!

It's the third letter on this problem - but I've got the same. When Sherlock
shows the list of items found, it says "The selected item could not be
found" and so on. However, MacOS 8.1 with standard Find dialog works fine.

Any comments? I've got MacOS 8.5.1, netatalk-1.4b2+asun2.1.3, linux 2.0.30.

--
Bye,
Paul S. Khristolubov,
Web developer, Publishing Firm of TPU
mailto:[email-masked] | http://www.ipf.tpu.ru/serp

----------
>From: [email-masked] (Jesus Ranz Abad)
>To: [email-masked]
>Subject: [netatalk-admins] Problems with sherlock searching netatalk
>Date: Tue, 16 Mar 1999 16:22:47 +0100

> When one of this clients (MacOS 8.5.1) try to find files using sherlook,
> the files are shows in the windows correctly but, if you try to click in
> some of then to see the path or to open it, the client say something like
> "It was  not possible to open the file because the file isn't there, maybe
> some one  have delete it". In this moment sherlook put the file like 'not
> accessible' changing the color name to gray.

---

From: a sun <[email-masked]>
Date: Tue, 16 Mar 1999 23:02:56 -0800 (PST)

It's the third letter on this problem - but I've got the same. When
   Sherlock shows the list of items found, it says "The selected item
   could not be found" and so on. However, MacOS 8.1 with standard
   Find dialog works fine.

read <http://til.info.apple.com/techinfo.nsf/artnum/n30845>. let's
just say that find file use to work in the pre-sherlock days, so i'm
not sure if it's actually netatalk's fault. 

-a

---

From: Rob Newberry <[email-masked]>
Date: Wed, 17 Mar 1999 11:07:55 -0500 (EST)

> not sure if it's actually netatalk's fault. 

Adrian,

I'm pretty certain that I know why Sherlock doesn't work.  I believe that
Sherlock index files for full text searches (and possibly how it tracks
"found" files) is done via file IDs.  If you think about it, that makes
sense, because using file IDs would keep index files much smaller.

So far, sherlock seems to work on AppleShare IP and NT servers.  BOTH of
these servers implement support for file IDs.  Sherlock doesn't work on
Novell or netatalk -- neither of them support file IDs.

Now that you have file ID support available, turn it on and see if
Sherlock magically starts working.  I'm pretty sure it will.

Note that this doesn't address a bigger issue about indexing netatalk
volumes.  Suppose several people have access to the volume, but each one
has different permissions.  Who gets to build the index?  If user A builds
it, then only that stuff that user A has access to will be indexed.  If
user B builds it, only the stuff that user B has access to will be
indexed.  Not to mention what happens when user B's auto-index building
time is set to some time after user A's, it might just overwrites user A's
index.  Now, obviously for individual user's "home" directories, this
won't be a problem -- supporting file IDs should allow them to index those
volumes right now.  But for shared directories, it would be coolest to
have the server somehow index the volume itself.  Unfortunately, this
requires knowlege of the format of those index files, which I don't
believe anyone except Apple possesses.

Rob

---

From: Rob Newberry <[email-masked]>
Date: Wed, 17 Mar 1999 12:17:31 -0500 (EST)

> i have two comments here:
>   1) sherlock doesn't use index files for find file searches. in fact,
>      you can't create text search index files right now. as i don't
>      have the sherlock find content db specs, i can't implement things
>      in afpd to create them either. in the future, i believe that
>      sherlock will be able to create index files on the server. in
>      that situation, of course, whoever created the file has the
>      rights to change it.

Just to be clear, I never said that Sherlock used index files for "found
files".  I meant that it might somehow use "file IDs" for found files. 
For "Find File", I know that Sherlock uses FBCatSearch if available, and
if not, does a recursive enumeration.  However, WHEN it finds a matching
file, it is still POSSIBLE that it resolves the file ID (using
FPCreateID), and stores that somewhere to reference when the user clicks
the item in the found list -- that's what I meant by saying "(and possibly
how it tracks "found"  files)".  If your experience shows that it doesn't
use file IDs in this situation, that's groovy.

Most likely, "Find File" in Sherlock uses the Alias Manager to create a
temporary alias record in memory to the found item, and stores that.  If
your volume supports file IDs, the alias manager will most likely use that
file ID to create the alias.  If not, it will create an alias using DID +
name, which SHOULD still work.

I am aware that you can't index things on the server because you don't
have the find content db specs.  I suspect that it's not an easy task
either -- otherwise, AIAT (which is what Sherlock uses) wouldn't have
taken Apple so long to produce.  It is conceivable that Apple will NEVER
document the format of these files, and consequently, we may never be able
to produce them independently.  This is doubtful, though -- someone could
probably reverse engineer the format of those files.  Still, until either
Apple documents it or someone figures it out, "afpd" obviously can't do
the indexing -- that's why I said it would be cool, but currently only
Apple possesses that knowlege.

Since the feature to index remote volumes has been disabled on the Mac,
we're hosed for content searches of afpd volumes until this information is
available.  I forgot that, so even user's home volumes are off limits for
Sherlock indexing right now.

I still think it's likely that file IDs are used in the index files for
purposes of reduced file sizes.  Otherwise, it would have to store entire
alias records.  Taking a quick look at a copy of TheFindByContentIndex on
my machine, I don't see things that look like alias records.  I'm still
betting on file IDs being necessary for full-text indexes.

But the fact that you can't index remote files certainly means we won't
know for a while :-).

>   2) let's just say that your "certainty" is misplaced. one of the
>      reasons why i added persistent did support was to see if it would
>      fix the sherlock problem. it doesn't. all it does is turn up
>      another problem which is most likely the underlying problem
>      causing sherlock to fail. actually, it turned out that all i
>      really had to do was pretend to add persistent dids to really
>      test things out.

I don't exactly understand this.  Netatalk has always "pretended" to have
persistent DIDs.  If I go do a "Find" and then click on an item, surely
the DID would have been persistent that long -- I haven't unmounted the
volume!  What is this "underlying problem"?  Is this the "incorrect
ordering of CNIDs that you're talking about?

> hmm. i did just have a thought. it may be that sherlock is incorrectly
> making some assumptions about the order the cnid's should follow. i'll
> go fiddle a little to see if that's it.

That seems pretty odd.  My guess would be that Sherlock creates an alias
record for each file that it finds, and stores a reference to that for
when the user clicks the item.  Creating an alias should resolve for items
if the DID is still the same.  Or, if you support file IDs, then the file
ID should be used by the Alias Manager, in which case, it should also
resolve.

Rob

---

From: a sun <[email-masked]>
Date: Wed, 17 Mar 1999 07:43:11 -0800 (PST)

-a

---

From: Brad Cox <[email-masked]>
Date: Wed, 17 Mar 1999 14:25:34 -0500

Do you have any ideas what could be causing this? I've tried 
everything I can think of: RTFM, downloading latest versions (but not 
your test directory version; yet), browsing the source, etc. This is 
on a Redhat 5.2 Intel box serving many macs, various OS versions 
(primary testing system running MacOS 8.5.1).

X-Sender: [email-masked]
Date: 	Wed, 17 Mar 1999 10:48:18 -0500
Reply-To: Appletalk on Linux <[email-masked]>
Sender: Appletalk on Linux <[email-masked]>
From: Brad Cox <[email-masked]>
Subject:      Still having spontaneous "server dropped connection" problems.
To: [email-masked]

Could someone please respond to this problem report? Thanks!

At 2:30 PM -0500 03/13/1999, Brad Cox wrote:
> With different first line.
>
> After running rock-solid for months, I've started getting messages to
> the effect that the linux file server has disappeared on the mac,
> whereupon the mounted files icons disappear from the desktop.
> Clicking an alias brings up a chooser window (with blank password),
> and filling that in remounts the icons just fine. If I have a file
> open on that filesystem with BBEdit and write it out, I get the same
> dialog and the file writes just fine.
>
> The only thing i've touched recently was my Netopia dual analog
> router, and that only to switch to a different phone number during a
> netcom pop outage. Doubt that can be it, but I've nothing else to go
> on.
>
> /var/log/messages contained these clues (note the "killed" lines).
> The others seem to have been created with I logged in again. Killed
> by what? And why? Any ideas? I can dig up further details if you tell
> me where and how to get them.
>
>> Mar 12 14:21:53 linux PAM_pwdb[2429]: (login) session opened for
>> user bcox by (u
>> id=0)
>> Mar 12 14:21:53 linux login[2429]: LOGIN ON ttyp2 BY bcox FROM g3.netcom.com
>> Mar 12 14:21:53 linux PAM_pwdb[2429]: (login) session closed for user bcox
>> Mar 12 19:22:21 linux afpd[288]: server_child[1] 2411 killed
>> Mar 12 19:22:28 linux afpd[2442]: ASIP session:548(1) from
>> 192.168.1.2:2062(2)
>> Mar 12 19:22:28 linux afpd[288]: server_child[1] 2442 done
>> Mar 12 19:22:31 linux afpd[2443]: ASIP session:548(1) from
>> 192.168.1.2:2063(2)
>> Mar 12 19:22:31 linux afpd[2443]: login bcox (uid 502, gid 100)
>> Mar 12 19:24:23 linux afpd[288]: server_child[1] 2443 killed
>> Mar 12 19:24:29 linux afpd[2444]: ASIP session:548(1) from
>> 192.168.1.2:2064(2)
>> Mar 12 19:24:29 linux afpd[288]: server_child[1] 2444 done
>> Mar 12 19:24:32 linux afpd[2445]: ASIP session:548(1) from
>> 192.168.1.2:2065(2)
>> Mar 12 19:24:32 linux afpd[2445]: login bcox (uid 502, gid 100)
>> Mar 12 14:36:39 linux PAM_pwdb[2454]: (su) session opened for user
>> root by bcox(
>> uid=0)
---
Dr. Brad Cox [email-masked] (update your address book)
Phone: 703 361 4751 Fax: 703 995 0422 Cellular: 703 919-9623
http://virtualschool.edu/mon A Project with Paradoxical Goals
PGP Signature: E194 C6E5 92D8 B8FB 20E8  8667 929A 95A0 FCB6 7C62
---
Dr. Brad Cox [email-masked] (update your address book)
Phone: 703 361 4751 Fax: 703 995 0422 Cellular: 703 919-9623
http://virtualschool.edu/mon A Project with Paradoxical Goals
PGP Signature: E194 C6E5 92D8 B8FB 20E8  8667 929A 95A0 FCB6 7C62

---

From: a sun <[email-masked]>
Date: Wed, 17 Mar 1999 12:01:52 -0800 (PST)

i guess i should have been clearer here. if you turn on volume level
file numbers, the mac will use FPCreateID, FPResolveID, and
FPDeleteID. all these are AFP 2.1 features, so sherlock shouldn't
really need them to work. now, to get FPResolveID to work, you really
need a database. however, it turns out that when you do a find file
with Sherlock, it asks you to resolve an ID of 0. that's an illegal
value to use. 

after some poking around, i did find out some information that seems
bizarre. it looks like sherlock is asking for information about the
root of the parent directory. as that doesn't really exist, i don't
think that's really allowed. i'll ask the powers that be what kind of
response sherlock might be expecting in such situations. it actually
exposed one of those "what do i do with this unexpected data" bugs in
netatalk.

with macos 8.5, the alias manager knows how to deal with changing
dids. in any case, everything is fixed during a single session, so
that shouldn't make a difference anyway.

-a

---

From: PayPC System Mail Subscriber <[email-masked]>
Date: Thu, 18 Mar 1999 12:30:40 +1000

Actually, it's not just netatalk that has problems....

LOCAL ISO9660 CD-ROM drives also do not function correctly when you're trying 
to search them with Sherlock.  Go ahead, try it.  I dare you. :)  So whatever 
Apple did with Sherlock, it broke just about everything that's NOT 
HFS/HFS+/Appleshare IP [and well stop my beating heart in surprise, NT 
Services for Macintosh].

Shame on Apple, really.  They obviously didn't test it all that much.

=R=

I may have missed a few e-mails, but...  now that I have netatalk purring 
smoothly [2.1.3 is a charm, Adrian]...  what is the hubbub about v2 versus v1 
Appledouble headers, et al?  What's the bonus to using v2?  What are the 
risks if I run "transparent" conversion?  Also.... I know there is early 
support for persistent DIDs and such, have people been using this much?  Does 
it freak out if permissions aren't writeable [cd-roms, or secure areas]?