Tcl Source Code

View Ticket
Login
Ticket UUID: 931847
Title: TclFSNormalizeToUniquePath - incomplete?
Type: Bug Created on: 2004-04-08 16:57:06
Submitter: dgp Assigned to: vincentdarley
Subsystem: 37. File System Severity:
Priority: 5 Medium Last modified: 2004-04-26 22:49:59
Status: Open Closed by:
Resolution: None Closed on:
Version: obsolete: 8.5a2
Description:
As I understand
TclFSNormalizeToUniquePath()
it's meant to force paths into
one canonical form -- getting
rid of symlinks, establishing
the case in the filesystem, etc.
To do this, it asks the filesystems
to do the work, giving the native
filesystems first try, then giving all
the others a try, calling the
Tcl_FSNormalizePathProc of
each, if implemented.

Each Tcl_FSNormalizePathProc(),
though, might not be able to finish
the job completely.  The "startAt"
value returned by TclFSNormalizeToUniquePath()
might not point to the end of the
returned string value of the path.

This is fine, as long as the callers
of TclFSNTUP() repeatedly call
it until startAt is at the end of the string,
or until nothing changes.  It appears
the existing callers of TclFSNTUP
are calling only once, not checking
the return value, and expecting
TclFSNTUP to have done all the
normalization possible.

The alternative to changing the callers,
is to rewrite TclFSNTUP itself to do
what the callers appear to expect,
keep looping over normalization
attempts until they don't do any more,
so the path is as normalized as possible.

This bug is difficult to demostrate
because it requires a non-native VFS
that actually implements a
Tcl_FSNormalizePathProc() that
would produce a changed path that,
say, the native filesystem could
normalize further.  For example,
imagine chaining symlinks into
and out of a non-native VFS.
User Comments:
dgp added on 2004-04-09 00:00:55:
Logged In: YES 
user_id=80530


A related issue:  The documentation
for new filesystems should more
explicitly warn that any 
Tcl_FSNormalizePathProc implementation
might be called on any path whatsoever,
not only the ones the filesystem thinks
belong to it.

dgp added on 2004-04-09 00:04:17:
Logged In: YES 
user_id=80530


Ack!  Expanding a symlink
might well introduce new
../ and ./ components to the path,
so perhaps the callers must
deal with this after all?

vincentdarley added on 2004-04-09 00:24:16:
Logged In: YES 
user_id=32170

Wow, you've been looking very closely!

I didn't design the code to handle symlinks between
filesystems, although I suppose I didn't actually document
that anywhere.  I certainly agree that that would
demonstrate bugs in the existing code.

In particular, however, the basic code assumption is that
the normalization can be achieved by (i) asking the native
filesystem to normalize, then (ii) asking each other
filesystem to normalize exactly once each.

Clearly if filesystems can be nested arbitrarily (e.g.
through your symlink approach), or more simply:

C:/foo/bar.zip/ZIP/foo.kit/KIT/blah.zip/ZIP

Assuming we have a separate .zip and .kit filesystem, each
of which have decided they are case-insensitive and that all
normalized paths should be lower case.  Then the above
should be normalized first by the OS (C:/foo/bar.zip) then
by the zipfs (/ZIP/foo.kit) then by the kitfs (KIT/blah.zip)
then by the zipfs again (ZIP).

In the current code structure this would not work.

I guess the correct solution is to first ask the nativefs,
and then, for the filesystem which accepts the path which
has been normalized so far, ask it to normalize some more. 
When no filesystem accepts the path, then just stop.  So,
we'd have:

Overall path accepted by native fs.
C:/foo/bar.zip   normalized by native, leaving 'startAt' at
following '/'

C:/foo/bar.zip/ recognised by zipfs
C:/foo/bar.zip/zip/foo.kit normalized by zipfs.

C:/foo/bar.zip/zip/foo.kit/ recognized by kitfs
etc.

So (pseudo-code):

while (1) {
    fs = PathInFilesystem(path, up to 'startAt');
    if (fs == null) break;
    startAt = fs->normalizePath(path, startAt);
}

with a bit of checking to avoid infinite loops.

This would even solve your symlink issue, I think, although
I haven't thought what it means for 'dir/..' handling.

dgp added on 2004-04-09 00:34:03:
Logged In: YES 
user_id=80530


FWIW, an arbitrarily nestable
filesystem is exactly what I'm
working on.

I think you've got the right basic
idea, on how to remove this limitation,
but there may be trouble with
PathInFilesystem().

Examining PathInFilesystem() is
what got me started.  This routine
gets passed an arbitrary path, and
must decide whether or not to claim
it.  Generally this can be a hard
problem, so it's natural to call
Tcl_FSGetNormalizedPath() on
the path passed in to have something
more predictable to work with.  That's
fine as long as a working Tcl_FSGNP
doesn't depend on a working PIF().
If so, we'll just pass the buck forever.

dgp added on 2004-04-09 00:39:46:
Logged In: YES 
user_id=80530


Now, I'm looking at
TclFSNormalizeAbsolutePath()
and it appears to have a lot
of overlapping function with
TclFSNormalizeToUniquePath().

In particular it appears to be
trying the same symlink removal.
that I understood TFNTUP to be
doing.

Any prospect for re-factoring some
of this so each operation is done
in only one place?

dgp added on 2004-04-09 01:51:50:
Logged In: YES 
user_id=80530

OK, I think I've assured myself
that Tcl_FSGetNormalizedPath()
is safe for the PathInFilesystem()
routine to call, so long as the
filesystem has its NormalizePath()
routine defined to handle any path
(if the filesystem has any normalizing
to do), and the Separator() routine
is defined.

vincentdarley added on 2004-04-09 16:31:26:
Logged In: YES 
user_id=32170

One relevant issue is that TFSNToUniquePath will only want
to call PathInFilesystem() for paths that it knows are
normalized up to a certain point, (and only wants to know if
the path up to that point is in the given fs).  I.e. in my
example below, we'll want to call
PathInFilesystem(C:/foo/bar.zip/), and we know that that
path is already normalized, which means we can use
'TclFSMakePathFromNormalized'   -(we don't actually want the
trailing '/', but that's a detail).

This means that we will actually want to construct a new (!)
pathPtr containing just this path, which we actually create
in the 'pure-normalized' form.  Unfortunately, the moment it
has been recognized, we throw it away.  This means in your
arbitrarily nested filesystem, normalization will be a
not-stunningly efficient process (but perhaps that doesn't
matter too much).

This suggests in the future we might want to extend the FS
interface to provide a more flexible pathInFilesystem proc
(Note: we have a private FS_VERSION_2 we can play with here).

Finally, concerning the overlap between the Absolute/Unique
routines, yes that is an issue.  I should clarify that the
Absolute routine is there _not_ to remove symlinks, but
rather to remove './' and '../' sequences (and this may
result in removing _some_ symlinks, if they are before a
'../' sequence).

Given your desire to go down the path of arbitrarily
nestable/symlinkable filesystems, I think a result of that
is we will have to refactor the two functions -- since, in
particular, '../' sequences can be re-introduced.

p.s. I'm largely away for the next week.

dgp added on 2004-04-26 22:49:59:
Logged In: YES 
user_id=80530

Reminder: when this gets
sorted out, look again at
Tcl Bug 941108.