Binary Extension

Note

This content was last updated August 1, 2023. Much of the content is tested automatically to keep from getting stale. If anything does not seem correct — or even if the explanation is insufficient — please file an issue.

The bezier Python package has optional speedups that wrap the libbezier library. These are incorporated into the Python interface via Cython as a binary extension. See C ABI (libbezier) for more information on building and installing libbezier.

Extra (Binary) Dependencies

When the bezier Python package is installed via pip, it will likely be installed from a Python wheel. The wheels uploaded to PyPI are pre-built, with the Fortran code compiled by GNU Fortran (gfortran). As a result, libbezier will depend on libgfortran. This can be problematic due to version conflicts, ABI incompatibility, a desire to use a different Fortran compiler (e.g. Intel’s ifort) and a host of other reasons.

There is standard tooling for distributing wheels that address this:

Linux: auditwheel
macOS: delocate
Windows: delvewheel

The tools address it by placing a copy of libgfortran (and potentially its dependencies) in the built wheel. This means that libraries that depend on libbezier may also need to link against these local copies of dependencies.

Linux

The command line tool auditwheel adds a bezier.libs directory to site-packages (i.e. it is next to bezier) with a modified libbezier and all of its dependencies (e.g. libgfortran)

>>> libs_directory
'.../site-packages/bezier.libs'
>>> print_tree(libs_directory)
bezier.libs/
  libbezier-f8ece041.so.2024.6.20
  libgfortran-040039e1.so.5.0.0
  libquadmath-96973f99.so.0.0.0

The bezier._speedup module depends on this local copy of libbezier:

$ readelf -d _speedup.cpython-312-x86_64-linux-gnu.so

Dynamic section at offset 0x47b000 contains 27 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../bezier.libs]
 0x0000000000000001 (NEEDED)             Shared library: [libbezier-f8ece041.so.2024.6.20]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x7000
 0x000000000000000d (FINI)               0x7cc20
...

and the local copy of libbezier depends on the other dependencies in bezier.libs/ (both directly and indirectly):

$ readelf -d ../bezier.libs/libbezier-f8ece041.so.2024.6.20

Dynamic section at offset 0x4adc8 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgfortran-040039e1.so.5.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libquadmath-96973f99.so.0.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libbezier-f8ece041.so.2024.6.20]
 0x000000000000000c (INIT)               0x3000
...
$ readelf -d ../bezier.libs/libgfortran-040039e1.so.5.0.0

Dynamic section at offset 0x275d78 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libquadmath-96973f99.so.0.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libgfortran-040039e1.so.5.0.0]
 0x000000000000000c (INIT)               0x19a88
...

Note

The runtime path (RPATH) uses $ORIGIN to specify a path relative to the directory where the extension module (.so file) is.

macOS

The command line tool delocate adds a bezier/.dylibs directory with copies of libbezier, libgfortran, libquadmath and libgcc_s:

>>> dylibs_directory
'.../site-packages/bezier/.dylibs'
>>> print_tree(dylibs_directory)
.dylibs/
  libbezier.2024.6.20.dylib
  libgcc_s.1.1.dylib
  libgfortran.5.dylib
  libquadmath.0.dylib

The bezier._speedup module depends on the local copy of libbezier:

$ otool -L _speedup.cpython-312-darwin.so
_speedup.cpython-312-darwin.so:
        @loader_path/.dylibs/libbezier.2024.6.20.dylib (...)
        /usr/lib/libSystem.B.dylib (...)

Though the Python extension module (.so file) only depends on libbezier it indirectly depends on libgfortran, libquadmath and libgcc_s:

$ otool -L .dylibs/libbezier.2024.6.20.dylib
.dylibs/libbezier.2024.6.20.dylib:
        /DLC/bezier/.dylibs/libbezier.2024.6.20.dylib (...)
        @loader_path/libgfortran.5.dylib (...)
        @loader_path/libquadmath.0.dylib (...)
        /usr/lib/libSystem.B.dylib (...)

Note

To allow the package to be relocatable, the libbezier dependency is relative to the @loader_path (i.e. the path where the Python extension module is loaded) instead of being an absolute path within the file system.

Notice also that delocate uses the nonexistent root /DLC for the install_name of libbezier to avoid accidentally pointing to an existing file on the target system.

Windows

The command line tool delvewheel adds a bezier.libs directory to site-packages (i.e. it is next to bezier) with a modified libbezier DLL

>>> libs_directory
'...\\site-packages\\bezier.libs'
>>> print_tree(libs_directory)
bezier.libs\
  bezier-40ff1ce7372f05ba11436ffbadd11324.dll
  libgcc_s_seh-1-5c71c85c0ca01174917203266ba98140.dll
  libgfortran-5-08073c6868a1df2cbc5609e49cbe3ad8.dll
  libquadmath-0-55d07eaa5b490be06911c864dcae60fd.dll
  libwinpthread-1-737bdf20e708783437e6fdbd7b05edf7.dll

The bezier._speedup module (.pyd file) depends on this local copy of libbezier:

> dumpbin /dependents _speedup.cp312-win_amd64.pyd
Microsoft (R) COFF/PE Dumper Version ...
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file _speedup.cp312-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    bezier-40ff1ce7372f05ba11436ffbadd11324.dll
    python312.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll

  Summary
...

and the local copy of libbezier depends on the other dependencies in bezier.libs/ (both directly and indirectly):

> dumpbin /dependents ..\bezier.libs\bezier-40ff1ce7372f05ba11436ffbadd11324.dll
Microsoft (R) COFF/PE Dumper Version ...
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file ..\bezier.libs\bezier-40ff1ce7372f05ba11436ffbadd11324.dll

File Type: DLL

  Image has the following dependencies:

    libgcc_s_seh-1-5c71c85c0ca01174917203266ba98140.dll
    libgfortran-5-08073c6868a1df2cbc5609e49cbe3ad8.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-private-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    KERNEL32.dll

  Summary
...

To enable building the Python binary extension, the libbezier DLL also has a corresponding import library — usr/lib/bezier.lib — which is provided to specify the symbols in the DLL.

On Windows, building Python extensions is a bit more constrained. Each official Python is built with a particular version of MSVC and Python extension modules must be built with the same compiler. This is primarily because the C runtime (provided by Microsoft) changes from Python version to Python version. To see why the same C runtime must be used, consider the following example. If an extension uses malloc from MSVCRT.dll to allocate memory for an object and the Python interpreter tries to free that memory with free from MSVCR90.dll, bad things can happen:

Python’s linked CRT, which is msvcr90.dll for Python 2.7, msvcr100.dll for Python 3.4, and several api-ms-win-crt DLLs (forwarded to ucrtbase.dll) for Python 3.5 … Additionally each CRT uses its own heap for malloc and free (wrapping Windows HeapAlloc and HeapFree), so allocating memory with one and freeing with another is an error.

This problem has been largely fixed in newer versions of Python but is still worth knowing.

Unfortunately, there is no Fortran compiler provided by MSVC. The MinGW-w64 suite of tools is a port of the GNU Compiler Collection (gcc) for Windows. In particular, MinGW includes gfortran. However, mixing the two compiler families (MSVC and MinGW) can be problematic because MinGW uses a fixed version of the C runtime (MSVCRT.dll) and this dependency cannot be easily dropped or changed.

Note

Although msvcrt.dll is a dependency of bezier.dll, it is not a problem. Any values returned from Fortran (as intent(out)) will have already been allocated by the caller (e.g. the Python process). This won’t necessarily be true for generic Fortran subroutines, but subroutines marked with bind(c) (i.e. marked as part of the C ABI of libbezier) will not be allowed to use allocatable or deferred-shape output variables. Any memory allocated in Fortran will be isolated within the Fortran code.

Source

For code that depends on libgfortran, it may be problematic to also depend on the local copy distributed with the bezier wheels.

The bezier Python package can be built from source if it is not feasible to link with these libraries, if a different Fortran compiler is required or “just because”.

The Python extension module can be built from source via:

$ # One of
$ BEZIER_INSTALL_PREFIX=.../usr/ python -m pip wheel .
$ BEZIER_INSTALL_PREFIX=.../usr/ python -m pip install .
$ BEZIER_INSTALL_PREFIX=.../usr/ python setup.py build_ext
$ BEZIER_INSTALL_PREFIX=.../usr/ python setup.py build_ext --inplace