exttypes Package

exttypes Package

attributetable Module

Extension attribute table type. Supports ordered (struct) fields, or unordered (hash-based) fields.

class numba.exttypes.attributetable.AttributeTable(py_class, parents)

Bases: object

Type for extension type attributes.

create_attribute_ordering(orderer=<function unordered at 0x377e7d0>)

Create a consistent attribute ordering with the base types.

ordering ∈ { unordered, extending, ... }

classmethod empty(py_class)
classmethod from_list(py_class, attributes)

Create a final attribute table from a list of attribute (name, type).

need_tp_dealloc()

Returns whether this extension type needs a tp_dealloc, tp_traverse and tp_clear filled out.

strtable()
to_struct()

autojitclass Module

Compiling @autojit extension classes works as follows:

  • Create an extension Numba/minivect type holding a symtab

  • Capture attribute types in the symtab in the same was as @jit

  • Build attribute hash-based vtable, hashing on (attr_name, attr_type).

    (attr_name, attr_type) is the only allowed key for that attribute (i.e. this is fixed at compile time (for now). This means consumers will always know the attribute type (and don’t need to specialize on different attribute types).

    However, using a hash-based attribute table allows easy implementation of multiple inheritance (virtual inheritance), without complicated C++ dynamic offsets to base objects (see also virtual.py).

For all methods M with static input types:
  • Compile M
  • Register M in a list of compiled methods
  • Build initial hash-based virtual method table from compiled methods

  • Create descriptors that wrap the native attributes

  • Create an extension type:

    {

    hash-based virtual method table (PyCustomSlots_Table **) PyGC_HEAD PyObject_HEAD ... native attributes

    }

    We precede the object with the table to make this work in a more generic scheme, e.g. where a caller is dealing with an unknown object, and we quickly want to see whether it support such a perfect-hashing virtual method table:

    if (o->ob_type->tp_flags & NATIVELY_CALLABLE_TABLE) {

    PyCustomSlots_Table ***slot_p = ((char *) o) - sizeof(PyGC_HEAD) PyCustomSlots_Table *vtab = **slot_p look up function

    } else {

    PyObject_Call(...)

    }

    We need to store a PyCustomSlots_Table ** in the object to allow the producer of the table to replace the table with a new table for all live objects (e.g. by adding a specialization for an autojit method).

class numba.exttypes.autojitclass.AutojitAttributeBuilder

Bases: numba.exttypes.compileclass.AttributeBuilder

create_descr(attr_name)

Create a descriptor that accesses the attribute on the ctypes struct. TODO: Use a perfect-hashed attribute table.

finalize(ext_type)
class numba.exttypes.autojitclass.AutojitExtensionCompiler(env, py_class, class_dict, ext_type, flags, method_maker, inheriter, method_filter, attrbuilder, vtabbuilder, methodwrapper)

Bases: numba.exttypes.compileclass.ExtensionCompiler

Compile @autojit extension classes.

exttype_validators = [<numba.exttypes.validators.AttributeTypeValidator object at 0x3956ad0>, <numba.exttypes.validators.MethodTypeValidator object at 0x3956b10>]
get_bases()

Get base classes for the resulting extension type.

We can try several inheritance schemes. We can choose between:

  1. One unspecialized - general - class

    • This must bind specialized methods on each object instance to allow method calls from Python, making object allocation more expensive
    • We must take the max() of tp_basicsize to allow enough space for all specializations. However, the specialization universe is not known up front, so we must allocate attributes separately on the heap (or perhaps we must override tp_alloc to use a dynamic size depending on the specialization - but this may conflict with non-numba base classes).
  2. One specialized class per instance

    • This allows us to experiment with static layouts for attributes or methods more easily
    • It seems more intuitive to back specialized objects with a specialized type

We will go with 2). We could go for a specialization tree as follows:

A

/ | A0 | A1

| |
B |
/ |

B0 B1

Which gets us:

issubclass(A_specialized, A) isinstance(A_specialized(), A)

as well as

issubclass(B_specialized, A_specialized)

However, to support this scheme, the unspecialized class A must:

  1. Be subclassable
  2. Return specialized object instances when instantiated
  3. Support unbound method calls

1) requires that A be a class, and then 2) implies that A’s metaclass overrides __call__ or that A implements __new__.

However, since A_specialized subclasses A, A_specialized.__new__ would need to skip A.__new__, which requires numba to insert a __new__ or modify a user’s __new__ method in A_specialized.

The metaclass option seems more feasible:

A_meta.__call__ -> specialized object instance

Users can then override a metaclass in a Python (or numba?) subclass as follows:

class MyMeta(type(MyNumbaClass)):
...

The metaclass can also support indexing:

A_specialized = A[{‘attrib_a’: double}]
get_metacls()
method_validators = [<numba.exttypes.validators.ArgcountMethodValidator object at 0x3956810>, <numba.exttypes.validators.InitMethodValidator object at 0x3956850>]
class numba.exttypes.autojitclass.AutojitMethodFilter

Bases: numba.exttypes.compileclass.Filterer

filter(methods, ext_type)
class numba.exttypes.autojitclass.AutojitMethodWrapperBuilder

Bases: numba.exttypes.compileclass.MethodWrapperBuilder

build_method_wrappers(env, extclass, ext_type)

Update the extension class with the function wrappers.

process_untyped_methods(env, extclass, ext_type)

Process autojit methods (undecorated methods). Use the fast NumbaSpecializingWrapper cache when for when we’re being called from python. When we need to add a new specialization, autojit_method_compiler is invoked to compile the method.

extclass: the extension type ext_type.py_class: the unspecialized class that was decorated

class numba.exttypes.autojitclass.UnboundDelegatingMethod(py_class, name)

Bases: object

Function in the unspecialized class that is used for delegation to a method in a specialized class, i.e.

A.method(A(10.0)) -> A(10.0).method()

This method can never be bound, since __new__ always returns specialized instances (so the unspecialized class cannot be instantiated!).

numba.exttypes.autojitclass.autojit_class_wrapper(py_class, compiler_impl, cache)

Invoked when a class is decorated with @autojit.

Parameters:
  • py_class – decorated python class
  • compiler_impl – compiler.ClassCompiler
  • cache – FunctionCache
Returns:

py_class that returns specialized object instances

numba.exttypes.autojitclass.autojit_method_compiler(env, extclass, method, signature)

Called to compile a new specialized method. The result should be added to the perfect hash-based vtable.

numba.exttypes.autojitclass.create_extension(env, py_class, flags, argtypes)

Create a partial environment to compile specialized versions of the extension class in.

Inovked when calling the wrapped class to compile a specialized new extension type.

numba.exttypes.autojitclass.make_delegations(py_class)

Make delegation unbound methods that delegate from the unspecialized class to the specialized class. E.g.

m = A.method m(A(10.0)) # Delegate to A[double].method

autojitmeta Module

Autojit meta class.

numba.exttypes.autojitmeta.create_specialized_metaclass(py_class)

When the autojit cache compiles a new class specialization, it invokes it with the constructor arguments. Since A_specialized inherits from A, AutojitMeta.__call__ again tries to specialize the class. We need to override this behaviour and instead instantiate A_specialized through type.__call__ (invoking A_specialized.{__new__,__init__}).

numba.exttypes.autojitmeta.create_unspecialized_cls(py_class, class_specializer)

Create an unspecialized class.

class_specializer:
NumbaSpecializingWrapper (args -> specialized object instance)

compileclass Module

class numba.exttypes.compileclass.AttributeBuilder

Bases: object

Build attribute descriptors for Python-level access.

build_descriptors(ext_type, extension_class)

Cram descriptors into the class dict

create_descr(attr_name)

Create a descriptor that accesses the attribute from Python space.

finalize(ext_type)

Finalize the attribute table (and fix the order if necessary)

class numba.exttypes.compileclass.AttributesInheriter

Bases: object

Inherit attributes and methods from parent classes:

For attributes and methods ...

  1. Build a table type
  2. Copy supertype slots into subclass table type
build_attribute_table(ext_type)
build_method_table(ext_type)
inherit(ext_type)

Inherit attributes and methods from superclasses

inherit_attributes(attr_table, base_ext_type)

Inherit attributes from a parent class. May be called multiple times for multiple bases.

inherit_methods(vtable, base_ext_type)

Inherit methods from a parent class. May be called multiple times for multiple bases.

class numba.exttypes.compileclass.ExtensionCompiler(env, py_class, class_dict, ext_type, flags, method_maker, inheriter, method_filter, attrbuilder, vtabbuilder, methodwrapper)

Bases: object

build_extension_type(vtable)

Build extension type from llvm methods and pointers and a populated virtual method table.

compile()

Compile extension methods:

  1. Process signatures such as @void(double)
  2. Infer native attributes through type inference on __init__
  3. Patch the extension type with a native attributes struct
  4. Infer types for all other methods
  5. Update the ext_type with a vtab type
  6. Compile all methods
compile_methods()

Compile all methods, reuse function environments from type inference stage.

∀ methods M sets M.lfunc, M.lfunc_pointer and M.wrapper_func

exttype_validators = None
finalize_tables()

Finalize (fix) the attribute and method tables.

get_bases()

Get base classes for the resulting extension type.

For jit types, these are simply the bases of the Python class we decorated. For autojit-decorated classes we get a more complicated inheritance scheme (see AutojitExtensionCompiler.get_bases).

get_metacls()

Return the metaclass for the specialized extension type.

infer()
  1. Infer extension attribute types from the __init__ method
  2. Type infer all methods
init()

Initialize:

  1. Inherit attributes and methods

    • Also build the vtab and attribute table types
  2. Process class attribute types:

    class Foo(object):

    myattr = double

  3. Process method signatures @void(double) etc

method_validators = None
process_method_signatures()

Process all method signatures:

  • Verify signatures
  • Populate ext_type with method signatures (ExtMethodType)
type_infer_init_method()
type_infer_method(method)
type_infer_methods()
validate()

Validate that we can build the extension type.

class numba.exttypes.compileclass.Filterer

Bases: object

filter(iterable, *args)
class numba.exttypes.compileclass.MethodWrapperBuilder

Bases: object

build_method_wrappers(env, extclass, ext_type)

Update the extension class with the function wrappers.

process_typed_methods(env, extclass, ext_type)
numba.exttypes.compileclass.build_extension_symtab(ext_type)

Create symbol table for all attributes of the extension type. These are Variables which are used by the type inferencer and used to type check attribute assignments.

New attribute assignments create new ExtensionAttributeVariable variables in the symtab. These variables update the attribute table during type inference:

class Foo(object):

value1 = double

def __init__(self, value2):
self.value2 = int_(value2)

Before type inference of __init__ we have:

symtab = { ‘value1’: Variable(double) }

and after type inference of __init__ we have:

symtab = {
‘value1’: Variable(double), # type is fixed ‘value2’: ExtensionAttributeVariable(int_), # type is inferred

}

numba.exttypes.compileclass.process_class_attribute_types(ext_type, class_dict)

Process class attribute types:

@jit class Foo(object):

attr = double

entrypoints Module

numba.exttypes.entrypoints.autojit_extension_class(env, py_class, flags, argtypes)

Compile an extension class given the NumbaEnvironment and the Python class that contains the functions that are to be compiled.

numba.exttypes.entrypoints.jit_extension_class(py_class, translator_kwargs, env)

jitclass Module

Compiling @jit extension classes works as follows:

  • Create an extension Numba/minivect type holding a symtab

  • Capture attribute types in the symtab ...

    • ... from the class attributes:

      @jit class Foo(object):

      attr = double

    • ... from __init__

      @jit class Foo(object):

      def __init__(self, attr):

      self.attr = double(attr)

  • Type infer all methods

  • Compile all extension methods

    • Process signatures such as @void(double)
    • Infer native attributes through type inference on __init__
    • Path the extension type with a native attributes struct
    • Infer types for all other methods
    • Update the ext_type with a vtab type
    • Compile all methods
  • Create descriptors that wrap the native attributes

  • Create an extension type:

    {

    PyObject_HEAD ... virtual function table (func **) native attributes

    }

The virtual function table (vtab) is a ctypes structure set as attribute of the extension types. Objects have a direct pointer for efficiency.

class numba.exttypes.jitclass.JitAttributeBuilder

Bases: numba.exttypes.compileclass.AttributeBuilder

create_descr(attr_name)

Create a descriptor that accesses the attribute on the ctypes struct. This is set by the extension type constructor __new__.

finalize(ext_type)
class numba.exttypes.jitclass.JitExtensionCompiler(env, py_class, class_dict, ext_type, flags, method_maker, inheriter, method_filter, attrbuilder, vtabbuilder, methodwrapper)

Bases: numba.exttypes.compileclass.ExtensionCompiler

Compile @jit extension classes.

exttype_validators = [<numba.exttypes.validators.AttributeTableOrderValidator object at 0x3956a50>, <numba.exttypes.validators.MethodTableOrderValidator object at 0x3956a90>, <numba.exttypes.validators.AttributeTypeValidator object at 0x3956ad0>, <numba.exttypes.validators.MethodTypeValidator object at 0x3956b10>]
method_validators = [<numba.exttypes.validators.ArgcountMethodValidator object at 0x3956750>, <numba.exttypes.validators.InitMethodValidator object at 0x3956790>, <numba.exttypes.validators.JitInitMethodValidator object at 0x39567d0>]
numba.exttypes.jitclass.create_extension(env, py_class, flags)

Compile an extension class given the NumbaEnvironment and the Python class that contains the functions that are to be compiled.

methodtable Module

Virtual method table types and ordering.

class numba.exttypes.methodtable.VTabType(py_class, parents)

Bases: object

Virtual method table type.

add_method(method)

Add a method to the vtab type and verify it with any parent method signatures.

create_method_ordering(orderer=<function unordered at 0x377e7d0>)

Create a consistent method ordering with the base types.

ordering ∈ { unordered, extending, ... }

classmethod empty(py_class)

Create an empty finalized vtable type

get_signature(method_name)

Get the signature for the given method name. Returns ExtMethodType

llvm_methods
method_pointers
methods

Return methods in the order they were set in

to_struct()

ordering Module

This module defines ordering schemes for virtual methods and attributes.

If we use hash-based virtual (method/attribute) tables, we don’t care about the ordering. If we’re using a C++ like virtual method/attribute table (like normal Python extension types do for attributes), we need to have a layout compatible with base classes (i.e. we may only add more attributes, but not reorder any existing ones).

class numba.exttypes.ordering.AbstractTable(table)

Bases: object

attrdict = None
attributes = None
parents
class numba.exttypes.ordering.AttributeTable(table)

Bases: numba.exttypes.ordering.AbstractTable

class numba.exttypes.ordering.VTable(table)

Bases: numba.exttypes.ordering.AbstractTable

numba.exttypes.ordering.extending(table)

Order the table entities according to the given parent tables, i.e. we can only extend existing tables.

numba.exttypes.ordering.sort_parents(table)

Sort parent tables by size

numba.exttypes.ordering.unordered(table)

Return table entities in a random order

numba.exttypes.ordering.validate_extending_order_compatibility(table)

signatures Module

Handle signatures of methods in @jit and @autojit classes.

class numba.exttypes.signatures.AutojitMethodMaker(argtypes)

Bases: numba.exttypes.signatures.MethodMaker

default_signature(method, ext_type)
class numba.exttypes.signatures.JitMethodMaker

Bases: numba.exttypes.signatures.MethodMaker

default_signature(method, ext_type)
no_signature(py_func)
class numba.exttypes.signatures.Method(py_func, name, signature, is_class, is_static, nopython=False)

Bases: object

py_func: the python ‘def’ function

clone()
get_wrapper()
update_from_env(func_env)
class numba.exttypes.signatures.MethodMaker

Bases: object

Creates Methods from python functions and validates user-declared signatures.

default_signature(method, ext_type)

Retrieve the default method signature for the given method if no user-declared signature exists.

make_method_type(method)

Create a method type for the given Method and declared signature

no_signature(method)

Called when no signature is found for the method

class numba.exttypes.signatures.MethodSignatureProcessor(class_dict, ext_type, method_maker, validators)

Bases: object

Processes signatures of extension types.

get_method_signatures()

Return [Method] for each decorated method in the class

update_signature(method)

Update a method signature with the extension type for ‘self’.

class Foo(object):
@void() # becomes: void(ext_type(Foo)) def method(self): ...
numba.exttypes.signatures.get_classmethod_func(func)

Get the Python function the classmethod or staticmethod is wrapping.

In Python2.6 classmethod and staticmethod don’t have the ‘__func__’ attribute.

numba.exttypes.signatures.has_known_signature(method)
numba.exttypes.signatures.method_argtypes(method, ext_type, argtypes)
numba.exttypes.signatures.process_signature(method, method_name, method_maker=<numba.exttypes.signatures.MethodMaker object at 0x39560d0>)

Verify a method signature.

Returns a Method object and the resolved signature. Returns None if the object isn’t a method.

utils Module

Simple utilities related to extension types

numba.exttypes.utils.get_all_numba_bases(py_class)
numba.exttypes.utils.get_class_dict(unspecialized_autojit_py_class)
numba.exttypes.utils.get_numba_bases(py_class)
numba.exttypes.utils.is_autojit_class(py_class)

Returns whether the given class is an unspecialized autojit class

numba.exttypes.utils.is_numba_class(py_class)

validators Module

Validate method signatures and inheritance compatiblity.

class numba.exttypes.validators.ArgcountMethodValidator

Bases: numba.exttypes.validators.MethodValidator

Validate a signature against the number of arguments the function expects.

validate(method, ext_type)

Validate a signature (which is None if not declared by the user) for a method.

class numba.exttypes.validators.AttributeTableOrderValidator

Bases: numba.exttypes.validators.ExtTypeValidator

Validate attribute table with static order (non-hash-based).

validate(ext_type)
class numba.exttypes.validators.AttributeTypeValidator

Bases: numba.exttypes.validators.ExtTypeValidator

Validate attribute types in the table with attribute types in the parent table.

E.g. if attribute ‘foo’ has type ‘double’ in the base class, then it should also have type ‘double’ in the derived class.

validate(ext_type)
class numba.exttypes.validators.ExtTypeValidator

Bases: object

Interface for validators that check for compatible inheritance trees.

validate(ext_type)

Validate an extension type with its parents.

class numba.exttypes.validators.InitMethodValidator

Bases: numba.exttypes.validators.MethodValidator

Validate the init method of extension classes.

validate(method, ext_type)
class numba.exttypes.validators.JitInitMethodValidator

Bases: numba.exttypes.validators.MethodValidator

Validate the init method for jit functions. Issue a warning when the signature is omitted.

check_init_args(method, ext_type)
validate(method, ext_type)
class numba.exttypes.validators.MethodTableOrderValidator

Bases: numba.exttypes.validators.ExtTypeValidator

Validate method table with static order (non-hash-based).

validate(ext_type)
class numba.exttypes.validators.MethodTypeValidator

Bases: numba.exttypes.validators.ExtTypeValidator

Validate method signatures in the vtable with method signatures in the parent table.

validate(ext_type)
class numba.exttypes.validators.MethodValidator

Bases: object

Interface for method validators

validate(method, ext_type)

Validate a Method. Raise an exception for user typing errors.

numba.exttypes.validators.validate_type_table(table, comparer)

Determine the compatability of this table with its parents given an ordering.AbstractTable and a type compare function ((type1, type2) -> bool).

variable Module

Extension attribute Variables used for attribute type inference. See also compileclass.build_extension_symtab().

class numba.exttypes.variable.ExtensionAttributeVariable(ext_type, attr_name, type, *args, **kwargs)

Bases: numba.symtab.Variable

Variable created during type inference for assignments to extension attributes for which we don’t know the type yet.

When the assignment happens, update ext_type.attributedict.

perform_assignment(rhs_type)

virtual Module

Virtual methods using virtual method tables.

Note that for @jit classes, we do not support multiple inheritance with incompatible base objects. We could use a dynamic offset to base classes, and adjust object pointers for method calls, like in C++:

However, this is quite complicated, and still doesn’t allow dynamic extension for autojit classes. Instead we will use Dag Sverre Seljebotn’s hash-based virtual method tables:

class numba.exttypes.virtual.HashBasedVTabBuilder

Bases: numba.exttypes.virtual.VTabBuilder

build_vtab(ext_type)
finalize(ext_type)
wrap_vtable(vtable)
class numba.exttypes.virtual.StaticVTabBuilder

Bases: numba.exttypes.virtual.VTabBuilder

build_vtab(ext_type)
finalize(ext_type)
wrap_vtable(vtable)
class numba.exttypes.virtual.VTabBuilder

Bases: object

Build virtual method table for quick calling from Numba.

build_vtab(ext_type, method_pointers)

Build a virtual method table. The result will be kept alive on the extension type.

finalize(ext_type)

Finalize the method table (and fix the order if necessary)

wrap_vtable(vtable)

Wrap the vtable such that users can get a pointer to the underlying data (extension_types.{Static,Dynamic}VtableWrapper).

numba.exttypes.virtual.build_hashing_vtab(vtable)

Build hash-based vtable.

numba.exttypes.virtual.build_static_vtab(vtable, vtab_struct)

Create ctypes virtual method table.

vtab_type: the vtab struct type (typesystem.struct) method_pointers: a list of method pointers ([int])

numba.exttypes.virtual.hash_signature(functype, name)
numba.exttypes.virtual.initialize_interner()
numba.exttypes.virtual.sep201_signature_string(functype, name)
numba.exttypes.virtual.vtab_name(field_name)

Mangle method names for the vtab (ctypes doesn’t handle this)