Doc: Add guide for adding extension modules to CPython
Adding a new C extension module to CPython's standard library requires updating multiple files across different build systems, which is not well documented. This adds a comprehensive guide that covers: - Prerequisites and design decisions (built-in vs shared) - Step-by-step process for all platforms - Unix/Linux: configure.ac and Setup file changes - Windows: MSBuild .vcxproj file creation - Testing, documentation, and cross-platform considerations - Troubleshooting common issues with solutions - Complete pre-submission checklist The guide is created in a new Doc/build_system/ directory for build-system-specific documentation that doesn't fit in the existing extending/ or c-api/ sections. This significantly reduces the barrier to adding new stdlib modules and provides a template for contributors to follow.
This commit is contained in:
Greg Shuflin
464
Doc/build_system/adding_modules.rst
Normal file
464
Doc/build_system/adding_modules.rst
Normal file
@@ -0,0 +1,464 @@
|
||||
.. _adding-stdlib-modules:
|
||||
|
||||
*************************************
|
||||
Adding Extension Modules to CPython
|
||||
*************************************
|
||||
|
||||
This guide explains how to add a new C extension module to the CPython standard
|
||||
library. It covers the complete process from initial setup through testing and
|
||||
cross-platform support.
|
||||
|
||||
.. note::
|
||||
|
||||
This guide is for adding modules to CPython's standard library (stdlib).
|
||||
For building third-party extension modules, see :ref:`extending-index`.
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
Adding a new extension module to CPython requires updating multiple files across
|
||||
the build system. The exact files depend on your target platforms, but typically
|
||||
include:
|
||||
|
||||
* Module source code (``.c`` and ``.h`` files in ``Modules/``)
|
||||
* Build configuration for Unix-like systems (``configure.ac``, Setup files)
|
||||
* Build configuration for Windows (MSBuild ``.vcxproj`` files)
|
||||
* Python interface and tests
|
||||
* Documentation
|
||||
|
||||
This guide focuses on the build system aspects. For guidance on writing the C
|
||||
code itself, see :ref:`extending-index` and :ref:`c-api-index`.
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
|
||||
Before adding a new module, you should:
|
||||
|
||||
* Be familiar with Python's C API and extension module development
|
||||
* Have a working CPython development environment
|
||||
* Understand your module's external dependencies (if any)
|
||||
* Know whether the module should be built-in (static) or shared (dynamic)
|
||||
|
||||
**Built-in vs. Shared Modules:**
|
||||
|
||||
* **Built-in (static)**: Linked directly into the Python interpreter. Used for
|
||||
essential modules like ``sys``, ``_io``, and ``_thread`` that are needed
|
||||
during interpreter startup.
|
||||
* **Shared (dynamic)**: Compiled as separate ``.so``/``.dylib``/``.pyd`` files.
|
||||
Most stdlib extension modules are shared. This is the recommended choice for
|
||||
new modules unless there's a specific reason to make them built-in.
|
||||
|
||||
Step-by-Step Guide
|
||||
===================
|
||||
|
||||
Step 1: Create Module Source Files
|
||||
-----------------------------------
|
||||
|
||||
Place your module source in the ``Modules/`` directory:
|
||||
|
||||
``Modules/mymodule.c``
|
||||
The main module implementation. Must include:
|
||||
|
||||
* A ``PyInit_mymodule`` function that creates and returns the module object.
|
||||
* Module methods, types, and other objects.
|
||||
* Module documentation string.
|
||||
|
||||
Minimal example::
|
||||
|
||||
#define PY_SSIZE_T_CLEAN
|
||||
#include "Python.h"
|
||||
|
||||
PyDoc_STRVAR(module_doc, "My new module.");
|
||||
|
||||
static PyMethodDef mymodule_methods[] = {
|
||||
{NULL, NULL, 0, NULL} /* Sentinel */
|
||||
};
|
||||
|
||||
static struct PyModuleDef mymodule = {
|
||||
PyModuleDef_HEAD_INIT,
|
||||
"mymodule", /* m_name */
|
||||
module_doc, /* m_doc */
|
||||
-1, /* m_size */
|
||||
mymodule_methods, /* m_methods */
|
||||
NULL, /* m_slots */
|
||||
NULL, /* m_traverse */
|
||||
NULL, /* m_clear */
|
||||
NULL, /* m_free */
|
||||
};
|
||||
|
||||
PyMODINIT_FUNC
|
||||
PyInit_mymodule(void)
|
||||
{
|
||||
return PyModule_Create(&mymodule);
|
||||
}
|
||||
|
||||
``Modules/mymodule.h`` (optional)
|
||||
Header file for internal declarations if the module is complex.
|
||||
|
||||
``Modules/clinic/mymodule.c.h`` (generated)
|
||||
If using Argument Clinic for parsing function arguments, this file will be
|
||||
generated. See :ref:`clinic-howto` for details.
|
||||
|
||||
Step 2: Configure for Unix-like Systems
|
||||
----------------------------------------
|
||||
|
||||
For Unix-like systems (Linux, macOS, BSD), you need to update autoconf
|
||||
configuration and Setup files.
|
||||
|
||||
2a. Update configure.ac
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Add dependency detection logic to ``configure.ac`` if your module requires
|
||||
external libraries. Find the section with other ``PY_STDLIB_MOD`` calls
|
||||
(around line 7800) and add::
|
||||
|
||||
# Example for a module with no dependencies
|
||||
PY_STDLIB_MOD([mymodule])
|
||||
|
||||
# Example for a module that requires libfoo
|
||||
PY_CHECK_LIB([foo], [foo_init], [LIBFOO_LIBS], [libfoo])
|
||||
PY_STDLIB_MOD([mymodule],
|
||||
[], [test "$LIBFOO_LIBS" != ""],
|
||||
[], [$LIBFOO_LIBS])
|
||||
|
||||
The ``PY_STDLIB_MOD`` macro parameters are:
|
||||
|
||||
1. Module name (as used in ``import mymodule``)
|
||||
2. Required dependencies (empty for none)
|
||||
3. Condition for building (shell test expression, leave empty for "always")
|
||||
4. Additional CFLAGS
|
||||
5. Additional LDFLAGS/LIBS
|
||||
|
||||
See existing modules in ``configure.ac`` for more examples.
|
||||
|
||||
After modifying ``configure.ac``, regenerate the configure script::
|
||||
|
||||
./Tools/build/regen-configure.sh
|
||||
|
||||
Or locally::
|
||||
|
||||
autoreconf -ivf -Werror
|
||||
|
||||
.. note::
|
||||
|
||||
The ``regen-configure.sh`` script uses a Docker container to ensure
|
||||
reproducible output with specific autoconf versions. The generated
|
||||
``configure`` script should be committed along with your changes to
|
||||
``configure.ac``.
|
||||
|
||||
2b. Update Modules/Setup.stdlib.in
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Add your module to ``Modules/Setup.stdlib.in`` in the appropriate section.
|
||||
For most new modules without dependencies::
|
||||
|
||||
@MODULE_MYMODULE_TRUE@mymodule mymodule.c
|
||||
|
||||
For modules with dependencies::
|
||||
|
||||
@MODULE_MYMODULE_TRUE@mymodule mymodule.c $(LIBFOO_CFLAGS) $(LIBFOO_LIBS)
|
||||
|
||||
The ``@MODULE_MYMODULE_TRUE@`` marker is substituted by the configure script:
|
||||
|
||||
* If dependencies are met, it becomes empty (module is included)
|
||||
* If dependencies are not met, it becomes ``#`` (module is commented out)
|
||||
|
||||
**Placement matters:**
|
||||
|
||||
* Modules before ``*static*`` are built-in (linked into the interpreter)
|
||||
* Modules after ``*shared*`` are shared libraries (typical for new modules)
|
||||
|
||||
For most new modules, add them in the shared section after the ``*shared*``
|
||||
marker.
|
||||
|
||||
Step 3: Configure for Windows
|
||||
------------------------------
|
||||
|
||||
Windows uses MSBuild and Visual Studio project files instead of autotools.
|
||||
|
||||
3a. Create a Visual Studio Project File
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Create ``PCbuild/mymodule.vcxproj`` based on an existing simple module's project
|
||||
file. For example, copy ``PCbuild/_csv.vcxproj`` and modify:
|
||||
|
||||
1. Update ``<ProjectName>`` to your module name
|
||||
2. Update source file paths in ``<ClCompile Include="...">`` sections
|
||||
3. Update dependencies in ``<ProjectReference>`` if needed
|
||||
4. Update preprocessor definitions if needed
|
||||
|
||||
Example structure::
|
||||
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<Project DefaultTargets="Build" ToolsVersion="4.0">
|
||||
<ItemGroup Label="ProjectConfigurations">
|
||||
<!-- Configuration entries (don't modify) -->
|
||||
</ItemGroup>
|
||||
<PropertyGroup Label="Globals">
|
||||
<ProjectGuid>{GENERATE-NEW-GUID-HERE}</ProjectGuid>
|
||||
<RootNamespace>mymodule</RootNamespace>
|
||||
</PropertyGroup>
|
||||
<Import Project="python.props" />
|
||||
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
|
||||
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
|
||||
<ItemGroup>
|
||||
<ClCompile Include="..\Modules\mymodule.c" />
|
||||
</ItemGroup>
|
||||
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
|
||||
</Project>
|
||||
|
||||
Generate a new GUID for your project::
|
||||
|
||||
python -c "import uuid; print('{' + str(uuid.uuid4()).upper() + '}')"
|
||||
|
||||
3b. Add to PCbuild/pcbuild.proj
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Add a reference to your new project in ``PCbuild/pcbuild.proj``::
|
||||
|
||||
<Projects Include="mymodule.vcxproj" />
|
||||
|
||||
This ensures your module is built as part of the standard Windows build.
|
||||
|
||||
3c. Update Windows Dependency Handling (if needed)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If your module depends on external libraries:
|
||||
|
||||
1. Check if the library is already in ``PCbuild/get_externals.bat``
|
||||
2. If not, add it to the external dependencies script
|
||||
3. Create appropriate property sheets (``.props`` files) for compiler/linker flags
|
||||
4. Reference the property sheets in your ``.vcxproj`` file
|
||||
|
||||
Step 4: Add Python-level Interface (Optional)
|
||||
----------------------------------------------
|
||||
|
||||
Many C extension modules have a Python wrapper in ``Lib/`` that provides a
|
||||
more Pythonic interface. For example:
|
||||
|
||||
* ``Modules/_json.c`` has ``Lib/json/``
|
||||
* ``Modules/_sqlite/`` has ``Lib/sqlite3/``
|
||||
* ``Modules/_csv.c`` has ``Lib/csv.py``
|
||||
|
||||
If your module provides a low-level C API, consider adding a Python wrapper
|
||||
in ``Lib/mymodule.py`` or ``Lib/mymodule/``.
|
||||
|
||||
Step 5: Add Tests
|
||||
-----------------
|
||||
|
||||
Create comprehensive tests in ``Lib/test/test_mymodule.py``::
|
||||
|
||||
import unittest
|
||||
import mymodule
|
||||
|
||||
class MyModuleTests(unittest.TestCase):
|
||||
def test_basic_functionality(self):
|
||||
# Test your module's functionality
|
||||
pass
|
||||
|
||||
def test_error_handling(self):
|
||||
# Test error cases
|
||||
pass
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
|
||||
Ensure tests cover:
|
||||
|
||||
* Basic functionality
|
||||
* Error cases and exceptions
|
||||
* Edge cases
|
||||
* Platform-specific behavior (if any)
|
||||
* Memory management (no leaks)
|
||||
|
||||
Run the test suite to verify::
|
||||
|
||||
./python -m test test_mymodule
|
||||
|
||||
Step 6: Update Documentation
|
||||
-----------------------------
|
||||
|
||||
Add or update documentation for your module:
|
||||
|
||||
``Doc/library/mymodule.rst``
|
||||
Full module documentation following the CPython documentation style.
|
||||
See existing module documentation for templates.
|
||||
|
||||
``Doc/library/index.rst``
|
||||
Add your module to the appropriate section of the library index.
|
||||
|
||||
``Doc/whatsnew/3.XX.rst``
|
||||
Document your new module in the "What's New" document for the next release.
|
||||
|
||||
Step 7: Build and Test
|
||||
-----------------------
|
||||
|
||||
Build CPython with your new module:
|
||||
|
||||
On Unix-like systems::
|
||||
|
||||
./configure
|
||||
make -j
|
||||
./python -c "import mymodule; print(mymodule.__doc__)"
|
||||
|
||||
On Windows::
|
||||
|
||||
PCbuild\build.bat
|
||||
PCbuild\amd64\python.exe -c "import mymodule; print(mymodule.__doc__)"
|
||||
|
||||
Run the full test suite::
|
||||
|
||||
make test
|
||||
|
||||
Or on Windows::
|
||||
|
||||
PCbuild\build.bat -t Test
|
||||
|
||||
Step 8: Handle Cross-platform Issues
|
||||
-------------------------------------
|
||||
|
||||
Testing on Multiple Platforms
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Your module should work on all supported platforms:
|
||||
|
||||
* **Linux**: Ubuntu, Debian, Fedora, etc. (different architectures: x86_64, ARM, etc.)
|
||||
* **macOS**: x86_64 and ARM64 (Apple Silicon)
|
||||
* **Windows**: x86_64 and ARM64
|
||||
* **WASM**: Emscripten and WASI (if applicable)
|
||||
* **Mobile**: iOS and Android (if applicable)
|
||||
|
||||
Platform-specific Code
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Use ``#ifdef`` guards for platform-specific code::
|
||||
|
||||
#ifdef MS_WINDOWS
|
||||
/* Windows-specific implementation */
|
||||
#elif defined(__APPLE__)
|
||||
/* macOS-specific implementation */
|
||||
#else
|
||||
/* Linux/BSD/other Unix */
|
||||
#endif
|
||||
|
||||
Or use ``Py_BUILD_CORE`` macros and features from ``pyconfig.h``.
|
||||
|
||||
Handling Missing Dependencies
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If your module's dependencies aren't available on all platforms:
|
||||
|
||||
1. Make it an **optional module** (not required for basic Python operation)
|
||||
2. Use ``PY_STDLIB_MOD`` conditions in ``configure.ac`` to disable on platforms
|
||||
without dependencies
|
||||
3. On Windows, use ``Condition`` attributes in ``.vcxproj`` to control building
|
||||
4. Document platform availability in module documentation
|
||||
|
||||
Checklist
|
||||
=========
|
||||
|
||||
Before submitting your module for review, verify:
|
||||
|
||||
Build System Integration:
|
||||
☐ Source files in ``Modules/``
|
||||
|
||||
☐ ``configure.ac`` updated with ``PY_STDLIB_MOD`` entry
|
||||
|
||||
☐ ``Modules/Setup.stdlib.in`` updated
|
||||
|
||||
☐ ``configure`` regenerated (``./Tools/build/regen-configure.sh``)
|
||||
|
||||
☐ Windows ``.vcxproj`` file created in ``PCbuild/``
|
||||
|
||||
☐ ``PCbuild/pcbuild.proj`` updated
|
||||
|
||||
Testing:
|
||||
☐ Test file created in ``Lib/test/``
|
||||
|
||||
☐ Tests pass on Linux
|
||||
|
||||
☐ Tests pass on macOS (if possible)
|
||||
|
||||
☐ Tests pass on Windows (if possible)
|
||||
|
||||
☐ No memory leaks (run with ``valgrind`` or ASAN)
|
||||
|
||||
Documentation:
|
||||
☐ Module documentation in ``Doc/library/``
|
||||
|
||||
☐ Library index updated
|
||||
|
||||
☐ What's New document updated
|
||||
|
||||
☐ Docstrings complete
|
||||
|
||||
Code Quality:
|
||||
☐ Follows :pep:`7` (C style guide)
|
||||
|
||||
☐ No compiler warnings
|
||||
|
||||
☐ Proper error handling (all error paths covered)
|
||||
|
||||
☐ Reference counting correct (no leaks or crashes)
|
||||
|
||||
Common Issues and Solutions
|
||||
============================
|
||||
|
||||
Module doesn't build on Linux
|
||||
------------------------------
|
||||
|
||||
**Problem**: After adding to ``Setup.stdlib.in``, module doesn't appear in build.
|
||||
|
||||
**Solution**:
|
||||
|
||||
1. Check that ``PY_STDLIB_MOD`` condition in ``configure.ac`` is satisfied
|
||||
2. Run ``./configure`` again to regenerate Makefile
|
||||
3. Check ``Modules/Setup.local`` doesn't override your module
|
||||
4. Examine configure output for your module's status
|
||||
5. Run ``make clean && make`` to force rebuild
|
||||
|
||||
Module builds but import fails
|
||||
-------------------------------
|
||||
|
||||
**Problem**: ``ImportError: dynamic module does not define module export function``
|
||||
|
||||
**Solution**:
|
||||
|
||||
Ensure your module initialization function is named exactly ``PyInit_modulename``
|
||||
where ``modulename`` matches the first argument to ``PyModule_Create`` and the
|
||||
filename.
|
||||
|
||||
Windows build fails
|
||||
-------------------
|
||||
|
||||
**Problem**: MSBuild can't find your module project or dependencies.
|
||||
|
||||
**Solution**:
|
||||
|
||||
1. Verify ``.vcxproj`` file XML is well-formed
|
||||
2. Ensure GUID is unique (not copied from another project)
|
||||
3. Check that ``PCbuild/pcbuild.proj`` includes your project
|
||||
4. Verify paths to source files are correct (use ``..`` to go up from ``PCbuild/``)
|
||||
5. Check that external dependencies are in ``PCbuild/`` after running ``get_externals.bat``
|
||||
|
||||
Module initialization crashes
|
||||
------------------------------
|
||||
|
||||
**Problem**: Python crashes during module import.
|
||||
|
||||
**Solution**:
|
||||
|
||||
1. Use a debugger (``gdb`` on Linux/macOS, Visual Studio debugger on Windows)
|
||||
2. Check reference counting (use ``--with-pydebug`` build)
|
||||
3. Verify all ``PyObject *`` pointers are checked for ``NULL``
|
||||
4. Ensure ``PyMODINIT_FUNC`` is used for the init function
|
||||
5. Build with ``--with-address-sanitizer`` to detect memory errors
|
||||
|
||||
See Also
|
||||
========
|
||||
|
||||
* :ref:`extending-index` - Extending Python with C or C++
|
||||
* :ref:`c-api-index` - Python/C API Reference
|
||||
* :pep:`7` - Style Guide for C Code
|
||||
* :ref:`clinic-howto` - Using Argument Clinic
|
||||
* `CPython Developer's Guide <https://devguide.python.org/>`_ - Full development workflow
|
||||
16
Doc/build_system/index.rst
Normal file
16
Doc/build_system/index.rst
Normal file
@@ -0,0 +1,16 @@
|
||||
.. _build-system-guides:
|
||||
|
||||
####################
|
||||
Build System Guides
|
||||
####################
|
||||
|
||||
These guides explain CPython's build system internals and how to work with it.
|
||||
|
||||
For basic build instructions, see :ref:`setup-building` in the `CPython Developer's Guide <https://devguide.python.org/>`_.
|
||||
|
||||
For configure options and build system overview, see :ref:`configure-options`.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
adding_modules.rst
|
||||
@@ -10,6 +10,7 @@
|
||||
reference/index.rst
|
||||
library/index.rst
|
||||
extending/index.rst
|
||||
build_system/index.rst
|
||||
c-api/index.rst
|
||||
installing/index.rst
|
||||
howto/index.rst
|
||||
|
||||
Reference in New Issue
Block a user