# TIP 500: Private Methods and Variables in TclOO
	Author:         Donal K. Fellows <donal.k.fellows@manchester.ac.uk>
	State:          Final
	Type:           Project
	Vote:           Done
	Created:        10-Feb-2018
	Post-History:
	Keywords:       Tcl, object orientation, visibility
	Tcl-Version:    8.7
	Votes-For:      DKF, JN, AF, JD, SL, AK, KBK
	Votes-Against:  none
-----

# Abstract

This TIP proposes a mechanism for private methods and variables in TclOO.
Private methods are methods that can only be called from methods of the same
class. Private variables have names so that they are unlikely to be used
by subclasses by accident (but can still be used from **vwait**, Tk, etc.)

# Rationale

One of the principles of object oriented programming is that classes should be
isolated from each other. This particularly includes the isolation of a
superclass (which might be in one package) from its subclasses (in other
packages) other than for its published API. The TclOO object system in Tcl 8.6
does not provide a way for user code to get such isolation: all classes that
are collaborating in a hierarchy to implement a particular instance object see
the same set of methods and variables. In the case of methods, it is because
the classes end up placing their methods into the same general resolution
scope (formed effectively from the union of all their methods), and in the
case of variables, it is because all variables are actually just normal Tcl
variables in the instance object's namespace.

A mechanism is desired to increase the isolation of classes from each other,
even where they are in a superclass-subclass relationship. Note that any such
mechanism does not need to be totally proof against prying (and introspection
mechanisms are desirable) as TclOO deliberately does not provide security
mechanisms; such mechanisms are the purpose of Tcl's interpreters. All that is
desired is a way of ensuring that code does not inadvertently trample other
code, allowing a superclass to evolve its implementation without needing to
take into account every detail of its subclasses.

# Private Declarations: Syntax and Immediate Semantics

This TIP introduces a new definition command, **private**, that is present
when doing both **oo::define** and **oo::objdefine**. Much as with the
**self** definition command (or the **oo::define** and **oo::objdefine**
commands themselves), it takes either single arguments or a list of arguments
(that it then builds a script of by turning them into a list) and evaluates
the result in a special context. The special context is the same as the outer
definition context, except that some definition commands instead create
_private definitions_ in that context.

The following commands are private-definition aware:

  * **forward** — the forwarded method it creates is a _private method_.
  * **method** — the procedure-like method it creates is a _private method_.
  * **self** (in **oo::class**) — the definitions on the class instance inside
    will be _private definitions_ (if the concept applies).
  * **variable** — the variable resolution rules created or manipulated (as
    this is a slot) will be for _private variables_.

If provided with no arguments, the **private** definition command will return
whether it was invoked from a context that should create private definitions.
Otherwise it returns the result of evaluating its script.

The meaning of a private method and a private variable is defined below.

# Detailed Proposal: Methods

For methods, as they actually exist in storage that is totally controlled by
TclOO (and which is not addressable other than via TclOO-created commands or
introspectors) the main complexity is working out what lookups are possible in
a particular case while ensuring that method lookups are fast. Determining
what the context class (or instance) is easy enough, as that's present pretty
much directly in the `Tcl_ObjectContext`, but rapidly determining whether the
object that is invoked is able to be seen in that way is the non-trivial part.
(As with most things in TclOO, appropriate caching is the key to accelerating
this sort of thing with a reasonable memory overhead.)

A key objective is that if one invokes `$otherobj ClassPrivateMethod` from
inside a method that is part of the class *but a different instance* or
possibly even a *different subclass* of that class, then that method will be
found. That makes this a useful mechanism that is not provided by the TclOO
system in Tcl 8.6.

## Method Invoke Semantics

When a method is invoked, a public (or unexported, if via the **my** command)
method is usually preferred. However, if the class of the object on which the
method is called is also the class that defined the currently executing method
(specifically, is the creator of the method that defines the current stack
frame at the point of the call), bearing in mind that the determination of
this also respects the inheritance and mixin hierarchies, then private methods
on that context class will also be found. Similarly, if the currently
executing method was defined by an object and that object is the object to
which the method invoke is directed, private methods on that object are also
found (the author expects these to be a rarely used feature). Searches for
private methods precede the equivalent public and unexported methods in the
method resolution order, but share a common naming space (i.e., they're stored
in the same hash tables) so that there are no ambiguities.

Since every method has at most one declaring class or object (and not both
simultaneously) there is at most one private method that can be on any call
chain. External calls (e.g., from the top level of Tcl, from a **namespace
eval**, from a procedure) will never have a private method on thir call
chains.
Once the call chain is created, the execution is handled as prior to this TIP;
the implementation of the first element on the call chain is executed,
whatever that is, and that can dispatch to later items on the call chain using
**next** and **nextto**.

A private method on a class (or object) may not have the same name as another
method on that class; all methods of a class (or object) are part of a single
space of names. Creating a public or unexported method with the same name in
the same declaration context will delete the private method, just as it would
delete any other visibility of method defined on the same scope; this is
assumed to be a minor issue for most code as the methods on a particular class
(as opposed to its superclasses, subclasses or instances) are assumed to be
strongly cooperative with each other.
When resolving a method call of a subclass of the class that
has the private method, even if the method names match, the private method
does not participate in the call chain; only an exact match from exactly the
right context counts.
Applying **export** or **unexport** to a private method will make that method
cease to be private, but it must be used on the same context (class or
instance) that defined the method; subclasses or instances cannot export or
unexport a private method defined by their superclass/class (respectively).
There is no mechanism for making an existing method private; that needs to be
done at method creation time.
Method declarations on other declaration contexts (e.g., on subclasses) do not
affect private methods at all. Private methods cannot be used for filter
implementations; the caller's context is not considered when resolving filters
so private methods are ignored (though a filter implementation may of course
call a private method).

In particular:

    oo::class create Top {
        method Foo {} {
            puts "This is Top::Foo for [self]"
        }
    }

    oo::class create Super {
        superclass Top

        private method Foo {} {
            puts "This is Super::Foo for [self]"
        }

        method bar {other} {
            puts "This is Super::bar for [self]"
            my Foo
            [self] Foo
            $other Foo
        }
    }

    oo::class create Sub {
        superclass Super

        method Foo {} {
            puts "This is Sub::Foo for [self]"
            next
        }

        private method bar {other} {
            error "This should be unreachable"
        }

        method grill {} {
            puts "This is Sub::grill for [self]"
            my Foo
        }
    }

    Sub create obj1
    Sub create obj2
    obj1 bar obj2
    obj1 grill

is expected to print out:

    This is Super::bar for ::obj1
    This is Super::Foo for ::obj1
    This is Super::Foo for ::obj1
    This is Super::Foo for ::obj2
    This is Sub::grill for ::obj1
    This is Sub::Foo for ::obj1
    This is Top::Foo for ::obj1

Note that the calls via **my**, **self** and through the public handle of
another object all pick up the private method from _Super_,
and the call to **next** from _Sub_ does not pick up the method on _Super_ but
instead goes straight to _Top_.

## Creation and Introspection

Private methods are created calling **oo::define** (and **oo::objdefine**, and
via the constructor of **oo::class**) like this:

    oo::define TheClass {
        private method foo {...} {
            ...
        }

        private forward bar  SomeOtherCommand
    }

At the C API level, private methods can be directly created by setting the
*flags* parameter (called *isPublic* in older versions of the API) of
`Tcl_NewMethod` and `Tcl_NewInstanceMethod` to the constant
`TCL_OO_METHOD_PRIVATE`. This lets custom types of methods be declared private
by their C code directly.

Introspection is done via appropriate options to **info class methods** and
**info object methods**. In particular, they are never returned when the
**-all** option is given, are found when **-private** option is given (unless
the **-all** option is _also_ given), and can be found when the new option
**-scope** is given, depending on what scope is chosen from the options below:

<dl><dt><b>-scope public</b></dt>
<dd>Reports the public (public) methods only.</dd>
<dt><b>-scope unexported</b></dt>
<dd>Reports the unexported (non-public, created via naming or <b>unexport</b>)
methods only.</dd>
<dt><b>-scope private</b></dt>
<dd>Reports the truly private methods only.</dd></dl>

The **-scope** option causes the **-all** and **-private** options to be
ignored.

# Detailed Proposal: Variables

Variables are more complex, conceptually, as they necessarily can be named by
other commands (e.g., so that they can be **vwait**ed upon or used with Tk
widgets; the key underlying operation in both of these cases is that they must
support traces) and so cannot be completely separated from the rest of Tcl.
This requires them to be named
by compounding the user-supplied name (to reduce confusion) and a unique
identifier that identifies the particular context of the name. Now, one of the
trickier things about TclOO is that it does not provide fixed names for
objects; the **rename** command is supported. Because of that, the
fully-qualified name of an object or class, while unique, is not a stable
identifier. The full name of the backing namespace *is* stably-named (as
namespaces cannot be renamed), but only has a unique name when the
fully-qualified name is used; single components of the name are not guaranteed
to be unique as users can control what those namespace names actually
are. However, there is also another unique identifier inside TclOO, the object
creation ID, which is based on a simple interpreter-level counter that is
incremented for each time an object is manufactured. (The creation ID is used
to allow very fast comparison between cached Tcl_Obj internal representations,
as well as being the seed for the automatically-created object names.) This
creation ID has a number of advantages, but its key ones are that it is
guaranteed to be a simple token (an integer!) and it is guaranteed to be
unique within an interpreter.

In particular, once a variable has been identified as being private by
declaration, it is visible to the methods of the class (or instance) that
defined it to be private with its given name, but it exists with a different
actual name in the namespace. That actual name (which is theoretically
globally visible once discovered) consists of the creation ID, a separator (a
string consisting of a space, a single colon, and another space) chosen to be
unlikely in any user script while also being disjoint from the syntax of
arrays, and the user-supplied portion of the name. String substitution using
these global names is _not recommended._ The variables concerned are not
created by declaration; that's up to an appropriate method to actually do (and
the variable can be created as normal scalar, array or link). When a method
runs in the relevant class, the class's variable resolver will map the private
variable in with the user-expected name.

The **variable** and **varname** methods of **oo::object** will be aware of
the mechanism, respecting what names are visible in the context that invokes
them. Thus, if **varname** is called from a context that would be able to see
the private variable, `foobar`, it will return the fully qualified name of the
Tcl variable variable in the target object's namespace that acts as the
storage of that variable; when the declaring class has creation ID `123` and
the instance of that class has namespace `::oo::Obj456`, the result of
`my varname foobar` will be `::oo::Obj456::123 : foobar`; the parts of that
name are:

 * “`::oo::Obj456`” — Fully-qualified instance namespace name; private
   variables are still ultimately variables that exist on object instances.
 * “`::`” — Namespace separator.
 * “`123`” — Declaring context ID, from the class that holds the definition of
   the private variable.
 * “<code> : </code>” — Private variable separator; specifically a space, a
   colon, and another space.
   This is chosen to be very unlikely to be in user scripts by accident (as
   variable names with spaces and single colons are traditionally known to be
   awkward to use explicitly).
 * “`foobar`” — User-supplied variable name fragment.

Similarly, the **variable** method can be used to bring the private variable
into scope; this would not normally be necessary, but is useful when the usual
resolution rules for the variable are suspended (such as when a variable with
that name is given as a formal argument to a constructor).

## Creation and Introspection

A private variable mapping is created like this:

    oo::define Foo {
        private variable x y

        method foo {} {
            set x 1
            set y(z) 2
        }
    }

That is, the **private** declaration command described above (for methods)
also impacts the **variables** slot declaration in **oo::define** and
**oo::objdefine**, causing it to introduce this more complex mapping
pattern. Note that the variable resolver already is class-scoped.
In the above example, a subclass of _Foo_ that refers to variables _x_ and _y_
will not see the variables defined by _Foo_, but will instead see either the
current standard variable scheme or their own _x_ and _y_. (If the creation ID
of _Foo_ is `123`, then _Foo_'s _x_ and _y_ will really be called `123 : x`
and `123 : y`, respectively.)

Introspection is via adding an extra option to **info class variables** and
**info object variables**, **-private**, which causes these commands to list
the private variables. Without the option, the non-private (i.e., direct
mapped) variables are listed.
A supporting introspector is also added, **info object creationid**, which
returns the creation ID of any existing object. It also applies to classes.
Again, note that creation IDs are _always_ system-allocated and are _never_
guaranteed to be unique between interpreters, either in multiple processes or
in the same thread or process; they are only ever locally unique.

# Examples

## Example: Private Variables and Tk

This example shows a private variable linked to a Tk entry:

    oo::class create Editable {
        variable w;            # Allow subclasses to see this
        private variable val;  # Hide this from subclasses

        constructor {widgetName} {
            set val {}
            set w [entry $widgetName -textvariable [my varname val]
        }

        method value {args} {
            if {[llength $args] == 0} {
                return $val
            } elseif {[llength $args] == 1} {
                lassign $args val
                return
            }
            return -code error "wrong # args: ..."
        }

        method trace {callbackLambda} {
            trace add variable val write [list apply $callbackLambda]
        } 
    }

    Editable create field .e
    field trace {args {
        puts "field is now [field value]"
    }}
    field value "Set the contents"

## Example: Private Methods and Code Complexity Management

Ths is an example of private methods. It shows how private methods can be used
to manage the complexity of a method without making its API (as exposed to
either subclasses or the rest of Tcl) overly complex.

    oo::class create Modulator {
        # The exported interface to the class
        method modulate {args} {
            try {
                foreach value $args {
                    my modulateOne $value
                }
            } on error {msg opts} {
                tailcall return {*}[my reworkError $msg $opts]
            }
        }

        private {
            # Do something with the single value; practical code would do more
            method modulateOne {value} {
                puts "this would modulate with $value"
                if {$value > 5} {
                    error "this is bad"
                }
            }

            # Simple way to hide the internal structure of the class from errors
            method reworkError {msg opts} {
                dict set opts -errorinfo "problem with modulating: $msg"
                list -options $opts $msg
            }
        }
    }

    set m [Modulator new]

    catch {$m modulate 4 5 6 7 8} -> opts
    puts [dict get $opts -errorinfo]
    # Prints something like:
    #    this would modulate with 4
    #    this would modulate with 5
    #    this would modulate with 6
    #    problem with modulating: this is bad
    #        invoked from within
    #    "::oo::Obj15 modulate 5 6 7 8"

    catch {$m no.such.method} msg
    puts $msg
    # Prints something like:
    #    unknown method "no.such.method": must be destroy or modulate

## Example: Private Methods and Variables Working Together

This is a combined example of private methods and variables.

    oo::class create LabelEqual {
        constructor {label} {
            set [my varname label] $label
        }

        private {
            # Two private declarations in the same block
            variable label

            method getLabel {} {
                return $label
            }
        }

        method equals {other} {
            # Note that this can call the private method of an
            # object other than itself. Private methods can make
            # class-internal protocols.
            expr {$label eq [$other getLabel]}
        }
    }

    oo::class create Evaluated {
        superclass LabelEqual

        # Poorly chosen variable name! Happens too easily in real life
        variable label

        constructor {expression} {
            next $expression
            set label [expr $expression]
        }

        method value {} {
            return $label
        }
    }

    set expr1 [Evaluated new {1 + 2 + 3}]
    set expr2 [Evaluated new {3 + 2 + 1}]

    puts "one is two? [$expr1 equals $expr2]"
    # Prints:
	#    one is two? 0

    puts "one=[$expr1 value] two=[$expr2 value]"
    # Prints:
	#    one=6 two=6

    puts [info vars [info object namespace $expr1]::*]
    # Prints something like:
	#    {::oo::Obj13::11 : label} ::oo::Obj13::label

## Example: Private Variable Naming and Creation IDs

This example highlights the behaviour of private variables and
`info object creationid`:

    # A simple introspection procedure for classes
    proc dumpinfo cls {
        puts "class ID of $cls is [info object creationid $cls]"
    }

    # A class with a private variable
    oo::class create Foo {
        private variable x

        constructor {} {
            # Remember, [self class] says what the class that declared
            # the current method is, i.e., ::Foo in this case.
            dumpinfo [self class]
            set x 1
        }

        method getPrivateX {} {
            # Returns the fully qualified name for 'x'
            return [my varname x]
        }
    }

    # A subclass with a normal variable with a potentially clashing name
    oo::class create Bar {
        superclass Foo

        constructor {} {
            # Ensure we call the superclass constructor first
            next

            dumpinfo [self class]
            set x 1
        }

        method getPublicX {} {
            return [my varname x]
        }
    }

    puts [Bar create abc]
    # Prints, for example:
    #    class ID of ::Foo is 11
    #    class ID of ::Bar is 12
    #    ::abc

    abc getPrivateX
    # Prints, for example:
    #    ::oo::Obj13::11 : x

    abc getPublicX
    # Prints, for example:
    #    ::oo::Obj13::x 

# Implementation

See the [`tip-500` branch](https://core.tcl.tk/tcl/timeline?r=tip-500).

# Copyright

This document has been placed in the public domain.
