PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by being defined with the keyword 'codef' instead of 'def'. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called directly from within the body of an ordinary function. An exception is raised if such a call to a cofunction is attempted. Cocalls ------- Inside the body of a cofunction, a function call such as :: f(*args, **kwds) is handled in a special way if f defines the special method ``__cocall__``. In this case, the above expression is evaluated as though it had been written :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. This kind of call will be referred to hereunder as a "cocall". If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement ``__cocall__`` are expected to return an object obeying the iterator protocol. Cofunctions respond to ``__cocall__`` the same way as ordinary generator functions respond to ``__call__``, i.e. by returning a generator-iterator. Cofunctions respond to __call__ by raising an exception, so that the error of attempting to call them from a non-cofunction is diagnosed clearly. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The only changes to the grammar are the introduction of a new keyword ``cocall`` and a minor variation on the syntax for defining a function: decorated: decorators (classdef | funcdef | cofuncdef) funcdef: 'def' NAME parameters ['->' test] ':' suite cofuncdef: 'codef' NAME parameters ['->' test] ':' suite New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Cofunctions address the first issue by making it possible to write calls to both ordinary functions and sub-coroutines using ordinary function call syntax. The second issue is addressed by making both kinds of call automatically work the right way when inside a cofunction, and by failing immediately if a cofunction is used incorrectly from an ordinary function. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Record of Discussion ==================== Implicit vs. Explicit Cocalls ----------------------------- When this proposal was first put forward, some respondents felt that the implicit yield-from behaviour of cocalls was too magical. It was suggested that the ability for any ordinary-looking function call to result in the suspension of one's coroutine would be too dangerous, and that it is important to be able to see exactly where the potential suspension points are. As a consequence, the second version of this proposal required a special syntax to be used when making cocalls, viz. :: cocall f(args) It was subsequently noticed that special syntax for marking both cofunction definitions and cocall sites was not necessary, and the third version of this proposal dropped the ``codef`` keyword, postulating that the presence of a ``cocall`` somewhere in the body would be sufficient to mark the function as being a cofunction, in the same way that the presence of ``yield`` marks a function as being a generator. However, by now the proposal had lost almost all of its original elegance and hardly addressed the original goal, i.e. making generator-based coroutine code just as easy and natural to write as any other code. Having to sprinkle ``cocall``s all over one's code is not much improvement over having to sprinkle ``yield from``s. On further reflection, the author has come to the view that the rationale put forward for disliking implicit cocalls is misguided. When dealing with concurrency in any form, if the correctness of your code is dependent on suspension only occurring at a few well-known points, then your code is brittle. Any change that adds a new suspension point requires re-evaluating all of the code to make sure that none of your earlier reasoning is invalidated. It is much better to start from the assumption that suspension can occur *anywhere*, except for a few small critical sections that you protect with some suitable form of synchronisation. With that mindset, the lack of an explicit marker for potentially-suspending call sites is of no consequence, while freedom from the burden of having to mark such sites is a considerable advantage. Alternatives to new syntax and novel magic ------------------------------------------ It has been questioned whether some combination of decorators and functions could be used instead of built-in mechanisms to achieve the same ends. While this might be possible, to achieve similar error-detecting power it would be necessary to write cofunction calls as something like :: yield from cofunc(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the present proposal using this kind of approach. Prototype Implementation ======================== An implementation of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: