TIP 500: Private Methods and Variables in TclOO

Login
Home Timeline Branches Tags Help 
Author:         Donal K. Fellows <[email protected]>
State:          Final
Type:           Project
Vote:           Done
Created:        10-Feb-2018
Post-History:
Keywords:       Tcl, object orientation, visibility
Tcl-Version:    8.7
Votes-For:      DKF, JN, AF, JD, SL, AK, KBK
Votes-Against:  none
Tcl-Branch:     tip-500

Abstract

This TIP proposes a mechanism for private methods and variables in TclOO. Private methods are methods that can only be called from methods of the same class. Private variables have names so that they are unlikely to be used by subclasses by accident (but can still be used from vwait, Tk, etc.)

Rationale

One of the principles of object oriented programming is that classes should be isolated from each other. This particularly includes the isolation of a superclass (which might be in one package) from its subclasses (in other packages) other than for its published API. The TclOO object system in Tcl 8.6 does not provide a way for user code to get such isolation: all classes that are collaborating in a hierarchy to implement a particular instance object see the same set of methods and variables. In the case of methods, it is because the classes end up placing their methods into the same general resolution scope (formed effectively from the union of all their methods), and in the case of variables, it is because all variables are actually just normal Tcl variables in the instance object's namespace.

A mechanism is desired to increase the isolation of classes from each other, even where they are in a superclass-subclass relationship. Note that any such mechanism does not need to be totally proof against prying (and introspection mechanisms are desirable) as TclOO deliberately does not provide security mechanisms; such mechanisms are the purpose of Tcl's interpreters. All that is desired is a way of ensuring that code does not inadvertently trample other code, allowing a superclass to evolve its implementation without needing to take into account every detail of its subclasses.

Private Declarations: Syntax and Immediate Semantics

This TIP introduces a new definition command, private, that is present when doing both oo::define and oo::objdefine. Much as with the self definition command (or the oo::define and oo::objdefine commands themselves), it takes either single arguments or a list of arguments (that it then builds a script of by turning them into a list) and evaluates the result in a special context. The special context is the same as the outer definition context, except that some definition commands instead create private definitions in that context.

The following commands are private-definition aware:

If provided with no arguments, the private definition command will return whether it was invoked from a context that should create private definitions. Otherwise it returns the result of evaluating its script.

The meaning of a private method and a private variable is defined below.

Detailed Proposal: Methods

For methods, as they actually exist in storage that is totally controlled by TclOO (and which is not addressable other than via TclOO-created commands or introspectors) the main complexity is working out what lookups are possible in a particular case while ensuring that method lookups are fast. Determining what the context class (or instance) is easy enough, as that's present pretty much directly in the Tcl_ObjectContext, but rapidly determining whether the object that is invoked is able to be seen in that way is the non-trivial part. (As with most things in TclOO, appropriate caching is the key to accelerating this sort of thing with a reasonable memory overhead.)

A key objective is that if one invokes $otherobj ClassPrivateMethod from inside a method that is part of the class but a different instance or possibly even a different subclass of that class, then that method will be found. That makes this a useful mechanism that is not provided by the TclOO system in Tcl 8.6.

Method Invoke Semantics

When a method is invoked, a public (or unexported, if via the my command) method is usually preferred. However, if the class of the object on which the method is called is also the class that defined the currently executing method (specifically, is the creator of the method that defines the current stack frame at the point of the call), bearing in mind that the determination of this also respects the inheritance and mixin hierarchies, then private methods on that context class will also be found. Similarly, if the currently executing method was defined by an object and that object is the object to which the method invoke is directed, private methods on that object are also found (the author expects these to be a rarely used feature). Searches for private methods precede the equivalent public and unexported methods in the method resolution order, but share a common naming space (i.e., they're stored in the same hash tables) so that there are no ambiguities.

Since every method has at most one declaring class or object (and not both simultaneously) there is at most one private method that can be on any call chain. External calls (e.g., from the top level of Tcl, from a namespace eval, from a procedure) will never have a private method on thir call chains. Once the call chain is created, the execution is handled as prior to this TIP; the implementation of the first element on the call chain is executed, whatever that is, and that can dispatch to later items on the call chain using next and nextto.

A private method on a class (or object) may not have the same name as another method on that class; all methods of a class (or object) are part of a single space of names. Creating a public or unexported method with the same name in the same declaration context will delete the private method, just as it would delete any other visibility of method defined on the same scope; this is assumed to be a minor issue for most code as the methods on a particular class (as opposed to its superclasses, subclasses or instances) are assumed to be strongly cooperative with each other. When resolving a method call of a subclass of the class that has the private method, even if the method names match, the private method does not participate in the call chain; only an exact match from exactly the right context counts. Applying export or unexport to a private method will make that method cease to be private, but it must be used on the same context (class or instance) that defined the method; subclasses or instances cannot export or unexport a private method defined by their superclass/class (respectively). There is no mechanism for making an existing method private; that needs to be done at method creation time. Method declarations on other declaration contexts (e.g., on subclasses) do not affect private methods at all. Private methods cannot be used for filter implementations; the caller's context is not considered when resolving filters so private methods are ignored (though a filter implementation may of course call a private method).

In particular:

oo::class create Top {
    method Foo {} {
        puts "This is Top::Foo for [self]"
    }
}

oo::class create Super {
    superclass Top

    private method Foo {} {
        puts "This is Super::Foo for [self]"
    }

    method bar {other} {
        puts "This is Super::bar for [self]"
        my Foo
        [self] Foo
        $other Foo
    }
}

oo::class create Sub {
    superclass Super

    method Foo {} {
        puts "This is Sub::Foo for [self]"
        next
    }

    private method bar {other} {
        error "This should be unreachable"
    }

    method grill {} {
        puts "This is Sub::grill for [self]"
        my Foo
    }
}

Sub create obj1
Sub create obj2
obj1 bar obj2
obj1 grill

is expected to print out:

This is Super::bar for ::obj1
This is Super::Foo for ::obj1
This is Super::Foo for ::obj1
This is Super::Foo for ::obj2
This is Sub::grill for ::obj1
This is Sub::Foo for ::obj1
This is Top::Foo for ::obj1

Note that the calls via my, self and through the public handle of another object all pick up the private method from Super, and the call to next from Sub does not pick up the method on Super but instead goes straight to Top.

Creation and Introspection

Private methods are created calling oo::define (and oo::objdefine, and via the constructor of oo::class) like this:

oo::define TheClass {
    private method foo {...} {
        ...
    }

    private forward bar  SomeOtherCommand
}

At the C API level, private methods can be directly created by setting the flags parameter (called isPublic in older versions of the API) of Tcl_NewMethod and Tcl_NewInstanceMethod to the constant TCL_OO_METHOD_PRIVATE. This lets custom types of methods be declared private by their C code directly.

Introspection is done via appropriate options to info class methods and info object methods. In particular, they are never returned when the -all option is given, are found when -private option is given (unless the -all option is also given), and can be found when the new option -scope is given, depending on what scope is chosen from the options below:

-scope public
Reports the public (public) methods only.
-scope unexported
Reports the unexported (non-public, created via naming or unexport) methods only.
-scope private
Reports the truly private methods only.

The -scope option causes the -all and -private options to be ignored.

Detailed Proposal: Variables

Variables are more complex, conceptually, as they necessarily can be named by other commands (e.g., so that they can be vwaited upon or used with Tk widgets; the key underlying operation in both of these cases is that they must support traces) and so cannot be completely separated from the rest of Tcl. This requires them to be named by compounding the user-supplied name (to reduce confusion) and a unique identifier that identifies the particular context of the name. Now, one of the trickier things about TclOO is that it does not provide fixed names for objects; the rename command is supported. Because of that, the fully-qualified name of an object or class, while unique, is not a stable identifier. The full name of the backing namespace is stably-named (as namespaces cannot be renamed), but only has a unique name when the fully-qualified name is used; single components of the name are not guaranteed to be unique as users can control what those namespace names actually are. However, there is also another unique identifier inside TclOO, the object creation ID, which is based on a simple interpreter-level counter that is incremented for each time an object is manufactured. (The creation ID is used to allow very fast comparison between cached Tcl_Obj internal representations, as well as being the seed for the automatically-created object names.) This creation ID has a number of advantages, but its key ones are that it is guaranteed to be a simple token (an integer!) and it is guaranteed to be unique within an interpreter.

In particular, once a variable has been identified as being private by declaration, it is visible to the methods of the class (or instance) that defined it to be private with its given name, but it exists with a different actual name in the namespace. That actual name (which is theoretically globally visible once discovered) consists of the creation ID, a separator (a string consisting of a space, a single colon, and another space) chosen to be unlikely in any user script while also being disjoint from the syntax of arrays, and the user-supplied portion of the name. String substitution using these global names is not recommended. The variables concerned are not created by declaration; that's up to an appropriate method to actually do (and the variable can be created as normal scalar, array or link). When a method runs in the relevant class, the class's variable resolver will map the private variable in with the user-expected name.

The variable and varname methods of oo::object will be aware of the mechanism, respecting what names are visible in the context that invokes them. Thus, if varname is called from a context that would be able to see the private variable, foobar, it will return the fully qualified name of the Tcl variable variable in the target object's namespace that acts as the storage of that variable; when the declaring class has creation ID 123 and the instance of that class has namespace ::oo::Obj456, the result of my varname foobar will be ::oo::Obj456::123 : foobar; the parts of that name are:

Similarly, the variable method can be used to bring the private variable into scope; this would not normally be necessary, but is useful when the usual resolution rules for the variable are suspended (such as when a variable with that name is given as a formal argument to a constructor).

Creation and Introspection

A private variable mapping is created like this:

oo::define Foo {
    private variable x y

    method foo {} {
        set x 1
        set y(z) 2
    }
}

That is, the private declaration command described above (for methods) also impacts the variables slot declaration in oo::define and oo::objdefine, causing it to introduce this more complex mapping pattern. Note that the variable resolver already is class-scoped. In the above example, a subclass of Foo that refers to variables x and y will not see the variables defined by Foo, but will instead see either the current standard variable scheme or their own x and y. (If the creation ID of Foo is 123, then Foo's x and y will really be called 123 : x and 123 : y, respectively.)

Introspection is via adding an extra option to info class variables and info object variables, -private, which causes these commands to list the private variables. Without the option, the non-private (i.e., direct mapped) variables are listed. A supporting introspector is also added, info object creationid, which returns the creation ID of any existing object. It also applies to classes. Again, note that creation IDs are always system-allocated and are never guaranteed to be unique between interpreters, either in multiple processes or in the same thread or process; they are only ever locally unique.

Examples

Example: Private Variables and Tk

This example shows a private variable linked to a Tk entry:

oo::class create Editable {
    variable w;            # Allow subclasses to see this
    private variable val;  # Hide this from subclasses

    constructor {widgetName} {
        set val {}
        set w [entry $widgetName -textvariable [my varname val]
    }

    method value {args} {
        if {[llength $args] == 0} {
            return $val
        } elseif {[llength $args] == 1} {
            lassign $args val
            return
        }
        return -code error "wrong # args: ..."
    }

    method trace {callbackLambda} {
        trace add variable val write [list apply $callbackLambda]
    } 
}

Editable create field .e
field trace {args {
    puts "field is now [field value]"
}}
field value "Set the contents"

Example: Private Methods and Code Complexity Management

Ths is an example of private methods. It shows how private methods can be used to manage the complexity of a method without making its API (as exposed to either subclasses or the rest of Tcl) overly complex.

oo::class create Modulator {
    # The exported interface to the class
    method modulate {args} {
        try {
            foreach value $args {
                my modulateOne $value
            }
        } on error {msg opts} {
            tailcall return {*}[my reworkError $msg $opts]
        }
    }

    private {
        # Do something with the single value; practical code would do more
        method modulateOne {value} {
            puts "this would modulate with $value"
            if {$value > 5} {
                error "this is bad"
            }
        }

        # Simple way to hide the internal structure of the class from errors
        method reworkError {msg opts} {
            dict set opts -errorinfo "problem with modulating: $msg"
            list -options $opts $msg
        }
    }
}

set m [Modulator new]

catch {$m modulate 4 5 6 7 8} -> opts
puts [dict get $opts -errorinfo]
# Prints something like:
#    this would modulate with 4
#    this would modulate with 5
#    this would modulate with 6
#    problem with modulating: this is bad
#        invoked from within
#    "::oo::Obj15 modulate 5 6 7 8"

catch {$m no.such.method} msg
puts $msg
# Prints something like:
#    unknown method "no.such.method": must be destroy or modulate

Example: Private Methods and Variables Working Together

This is a combined example of private methods and variables.

oo::class create LabelEqual {
    constructor {label} {
        set [my varname label] $label
    }

    private {
        # Two private declarations in the same block
        variable label

        method getLabel {} {
            return $label
        }
    }

    method equals {other} {
        # Note that this can call the private method of an
        # object other than itself. Private methods can make
        # class-internal protocols.
        expr {$label eq [$other getLabel]}
    }
}

oo::class create Evaluated {
    superclass LabelEqual

    # Poorly chosen variable name! Happens too easily in real life
    variable label

    constructor {expression} {
        next $expression
        set label [expr $expression]
    }

    method value {} {
        return $label
    }
}

set expr1 [Evaluated new {1 + 2 + 3}]
set expr2 [Evaluated new {3 + 2 + 1}]

puts "one is two? [$expr1 equals $expr2]"
# Prints:
#    one is two? 0

puts "one=[$expr1 value] two=[$expr2 value]"
# Prints:
#    one=6 two=6

puts [info vars [info object namespace $expr1]::*]
# Prints something like:
#    {::oo::Obj13::11 : label} ::oo::Obj13::label

Example: Private Variable Naming and Creation IDs

This example highlights the behaviour of private variables and info object creationid:

# A simple introspection procedure for classes
proc dumpinfo cls {
    puts "class ID of $cls is [info object creationid $cls]"
}

# A class with a private variable
oo::class create Foo {
    private variable x

    constructor {} {
        # Remember, [self class] says what the class that declared
        # the current method is, i.e., ::Foo in this case.
        dumpinfo [self class]
        set x 1
    }

    method getPrivateX {} {
        # Returns the fully qualified name for 'x'
        return [my varname x]
    }
}

# A subclass with a normal variable with a potentially clashing name
oo::class create Bar {
    superclass Foo

    constructor {} {
        # Ensure we call the superclass constructor first
        next

        dumpinfo [self class]
        set x 1
    }

    method getPublicX {} {
        return [my varname x]
    }
}

puts [Bar create abc]
# Prints, for example:
#    class ID of ::Foo is 11
#    class ID of ::Bar is 12
#    ::abc

abc getPrivateX
# Prints, for example:
#    ::oo::Obj13::11 : x

abc getPublicX
# Prints, for example:
#    ::oo::Obj13::x

Implementation

See the tip-500 branch.

Copyright

This document has been placed in the public domain.