Python Nested Lexical Scoping Enhancement

Greg Ewing, August 1999

This enhancement adds full nested lexical scoping to Python 1.5.2 in a way which avoids the worst of the cyclic data structure problems which prevented earlier lexical scoping attempts from catching on.

Download

lexscope.tar.gz
8643 bytes, last updated 9 Aug 1999

What it provides

Nested functions and lambda expressions can read the local variables of lexically enclosing functions and lambdas.

What it doesn't provide

Currently, intermediate variable access is read-only; if you want to change the value of an intermediate variable, you'll have to wrap it in a mutable container. Write access to intermediate variables would not be difficult to implement, but it would require a grammar change, and I haven't decided how best to do it without requiring a new keyword.

How it works

Stack frames have a new field, f_staticlink, which is null for top level function invokations and points to the lexically enclosing frame for nested function invokations. Various parts of ceval.c have been modified to pass around static link information where required.

Whenever a non-local variable is loaded (using the LOAD_GLOBAL opcode)  the chain of static links is searched, starting one stack frame up from the current frame, for a local variable by that name. If none is found, the global environment is accessed as usual. (For completeness, LOAD_NAME also does this starting from the current stack frame.)

There is a new callable object type, PyClosureObject, having two attributes:

function
A PyFunctionObject.

 
environment
A PyFrameObject which is used as the static link when the closure is called.
There is a new opcode MAKE_CLOSURE which takes a function object and constructs a closure object from it and the current stack frame. This opcode is emitted by the compiler after the MAKE_FUNCTION opcode of a lambda expression.

To avoid creating loops among stack frames when defining nested functions, execution of a nested def does not immediately create a closure object. Instead, it creates another new object type, PyLocalFunctionObject, which simply wraps the function object, and stores that in the local variable. Another new opcode, MAKE_LOCAL_FUNCTION, is used for this.

Whenever a value of type PyLocalFunction is loaded from a local variable by the LOAD_FAST (or LOAD_NAME) opcode, a PyClosure is created from the function and the stack frame in which it was found. The closure is then returned as if it had been the value found in the local variable.

Shortcomings

It is still possible to create cycles if you're not careful. For instance, storing a local function or the value of a lambda expression in a local variable of a scope enclosing its definition will create a cycle.

You can also create cycles among instances by plugging local functions or lambdas into them as callbacks.

I don't think these are any worse than the usual sort of cycle problems, however, and the remedy is the same - make sure you break all the cycles you create. The only difference is that you will have a few more ways of creating cycles than you do in regular Python!