Source Files and Compilation

File Names and Extensions
Modules in Packages
Building an Extension
Distributing Pyrex modules

File Names and Extensions

Pyrex source file names consist of the name of the module followed by a .pyx extension, for example a module called primes would have a source file named primes.pyx.

Modules in Packages

If your module is destined to live in a package, the Pyrex compiler needs to know the fully-qualified name that the module will eventually have.

There are currently two ways to give it this information:

Name the source file with the full dotted name of the module. For example, a module called primes to be installed in a package called numbers would have a source file called numbers.primes.pyx.
Place the source file in a package directory. To Pyrex, a package directory is one that contains a file called either __init__.py or __init__.pyx. For example, a package called numbers containing a module called primes would have the source files laid out like this:

numbers

__init__.py
primes.pyx

This will ensure that the __name__ properties of the module and any classes defined in it are set correctly. If you don't do this, you may find that pickling doesn't work, among other problems. It also ensures that the Pyrex compiler has the right idea about the layout of the module namespace, which can be important when accessing extension types defined in other modules.

Building an Extension

There are two steps involved in creating an extension module from Pyres sources:

Use the Pyrex compiler to translate the .pyx file into a .c file.
Compile the .c file with a C compiler and link it with whatever libraries it needs, to produce an extension module.

There are a variety of ways of accomplishing these steps, either separately or together. One way is to compile the Pyrex source manually from the command line with the Pyrex compiler, e.g.

pyrexc -r primes.pyx

This will compile primes.pyx and any other source files that it depends on, if any of them have changed since the last compilation, and produce a file called primes.c, which then needs to be compiled with the C compiler using whatever options are appropriate on your platform for generating an extension module.

You can perform the C compilation using distutils and a setup.py file, or with a conventional Makefile. There's a Makefile in the Demos directory (called Makefile.nodistutils) that shows how to do this for Linux.

Another approach is to put code at the beginning of your setup.py file to import the Pyrex compiler and call it from Python. You can then follow this with a normal call to setup() to compile the resulting .c files.

You can also perform both steps at once in a setup.py file using the distutils extension provided with Pyrex. See the Setup.py file in the Demos directory for an example of how to use it. A disadvantage of this method is that you won't be able to take advantage of Pyrex's own dependency checking features to compile only the Pyrex sources which have changed.

Command Line

You can run the Pyrex compiler from the command line using either the pyrexc shell command or the Python version of it, pyrexc.py.

The following command line options exist:

Short	Long	Description
-v	--version	Display the version number of the Pyrex compiler
-l	--create-listing	Produces a .lis file for each compiled .pyx file containing error messages
-I	--include-dir	Specifies a directory to be searched for included files and top-level package directories. Multiple -I options may be given, each specifying one directory.
-o	--output-file	Specifies name of generated C file. Only meaningful when a single .pyx file is being compiled.
-r	--recursive	Compile the given .pyx files, plus those of any modules it depends on directly or indirectly via cimport statements. The include path specified by -I options is used to find the .pyx files of dependent modules.
-t	--timestamps	Use modification times of files to decide whether to compile a .pyx file. This is the default when -r is used, unless -f is also used.
-f	--force	Compile all .pyx files regardless of modification times. This is the default when -r is not given.
-q	--quiet	When -r is given, don't display the names of source files being compiled.

Calling the Pyrex compiler from Python

The module Pyrex.Compiler.Main exports the following classes and functions to facilitate invoking the compiler from another Python program.

compile(source [, options] [, option = value ]...)

Compiles one or more Pyrex source files, which should be .pyx files. The source argument may be either a single filename or a list of filenames.

Depending on the recursive option, it may compile just the specified source files, or the specified source files plus those of other modules that they depend on via cimport statements. The options may be given either as keyword arguments or a CompilationOptions instance. If both are used, keyword arguments take precedence.

The return value depends on whether a list of sources was specifed and whether the recursive option is in effect. If a single source file is specified and the recursive option is false, the return value is a CompilationResult instance. Otherwise, the return value is a CompilationResultSet containing a CompilationResult for each of the modules which were actually compiled (which may or may not include ones corresponding to the specified source files).

Note: If you have more than one source file to compile, it is more efficient to do so with a single call to compile rather than one call for each source file. This is because, if more than one source cimports the same .pxd file, the .pxd file is parsed only once instead of being parsed each time it is cimported.

compile_single(source_path [, options] [, option = value ]...)

Compiles just a single .pyx source file, specified as a string, with no dependency or timestamp checking (the recursive and timestamps options are ignored). Always returns a CompilationResult.

compile_multiple(source_list [, options] [, option = value ]...)

Compiles a list of .pyx source files, with optional dependency and timestamp checking as for compile. Always takes a list of source pathnames, and always returns a CompilationResultSet.

class CompilationOptions

A collection of options to be passed to the compile() function. The following options may be specified as keyword arguments to either the CompilationOptions constructor or the compile() function.

show_version: Display the version number of the Pyrex compiler.
use_listing_file: Produce a .lis file for each .pyx file containing compilation errors.
include_path: A list of directories to search for included files and top-level package directories.
output_file: Use the given name for the generated .c file (only in non-recursive mode).
recursive: Compile the .pyx files of any cimported modules in addition to the one specified.
timestamps: Only compile modified sources as determined by file modification times. This may be true, false or None. If None, timestamps are used if and only if recursive mode is in effect.
quiet: Don't display names of sources being compiled in recursive mode.

class CompilationResult

An object providing information about the result of compiling a .pyx file. It has the following attributes:

c_file: Pathname of the generated C source file.
h_file: Pathname of the generated C header file, if any.
api_file: Pathname of the generated C API header file, if any.
listing_file: Pathname of the generated error message file, if any.
num_errors: Number of compilation errors.

class CompilationResultSet

This is a mapping object whose keys are the pathnames of .pyx files and whose values are the corresponding CompilationResult instances. It also has the following additional attributes:

num_errors: Total number of compilation errors.

Using the distutils extension

The distutils extension provided with Pyrex allows you to pass .pyx files directly to the Extension constructor in your setup file.

To use it, you need to put the following at the top of your setup.py file:

from Pyrex.Distutils.extension import Extension
from Pyrex.Distutils import build_ext

and you need to specify the Pyrex version of the build_ext command class in your setup() call:

setup(
    ...
    cmdclass = {'build_ext': build_ext}
)

Using the distutils extension is not currently recommended, because it's unable to automatically find cimported modules or check the timestamps of .pxd files and included sources. It also makes it harder to turn off the use of Pyrex for people who are only installing your module and not modifying the sources.

Distributing Pyrex modules

It is strongly recommended that you distribute the generated .c files as well as your Pyrex sources, so that users can install your module without needing to have Pyrex available.

It is also recommended that Pyrex compilation not be enabled by default in the version you distribute. Even if the user has Pyrex installed, he probably doesn't want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.

---