From Publishing Python Packages by Dane Hillard
This article covers the pieces of the Python build system itself, so you can learn how packages are built from scratch.
Take 35% off Publishing Python Packages by entering fcchillard2 into the discount code box at checkout at manning.com.
Python package builds are the product of coordination between a few different tools driven by a standardized process. One of the biggest choices you have as a package author is which set of tools to use. It can be difficult to assess the nuances of each, especially if you’re new to packaging. Fortunately, tools are standardizing around the same core workflow, so once you learn it you’ve got the agility to switch between tools with minimal effort.
This article covers what you first need to learn about the pieces of the Python build system itself.
To follow along with this article, you will need to install and use build. If you haven’t done this before, see the steps below.
build (github.com/pypa/build) is a tool provided by the Python Packaging Authority (PyPA) for building Python packages.
Install build using the following command:
python -m pip install build
To verify your configuration, run the following command:
$ python -m build --version
In the root directory for your package, start by running
build using the following command:
$ python -m build
Because your package has no content yet, you should see an error like the following:
ERROR Source /Users/<you>/code/first-python-package does not appear to be a Python project: no pyproject.toml or setup.py
The output makes two file suggestions.
pyproject.toml is the newer standard file for configuring Python packaging introduced in PEP 518 (https://www.python.org/dev/peps/pep-0518/), and should be preferred unless a third-party tool you want to use is only compatible with
setup.py. The file uses TOML (https://toml.io/en/), an INI-like language, to split configuration into relevant sections. Create the
pyproject.toml file using the following command to correct the error:
$ touch pyproject.toml
build command again. This time the build should run successfully, and you should see a large amount of output with a few notable lines as shown in listing 1. What’s happening here? At a high level, the build command consumes your source code and the metadata you supply, along with some files it generates, to create:
- A source distribution package. A Python source distribution, or
sdist, is a compressed archive file of the source code with a
- A binary distribution package. A Python built distribution package is a binary file. The current standard for built distributions is what’s known as a wheel or
bdist_wheel, a file with a
Listing 1. The result of building an empty Python package
... Successfully installed setuptools-57.0.0 wheel-0.36.2 ❶ ... running sdist ❷ ... warning: sdist: standard file not found: ❸ should have one of README, README.rst, README.txt, README.md running check warning: check: missing required meta-data: name, url ❹ warning: check: missing meta-data: either (author and author_email) ❺ or (maintainer and maintainer_email) should be supplied creating UNKNOWN-0.0.0 ❻ ... Creating tar archive ❼ ... Successfully installed setuptools-57.0.0 wheel-0.36.2 ... running bdist_wheel ❽ ... creating '/Users/<you>/code/first-python-package/dist/tmpgdfzly_7/ ❾ UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build/bdist.macosx-11.2-x86_64/wheel' to it
❶ Setuptools and the
wheel package are used for the build backend
❷ The source distribution package is built by the
❸ The build process expects a
README file in one of a few formats
❹ The build process expects a name and a URL for the package
❺ The build process expects an author or maintainer of the package
❻ The package is called
UNKNOWN because no name was specified
❼ The source distribution is a compressed archive file
❽ The binary wheel distribution package is built by the
❾ The binary wheel distribution is a
Because you haven’t supplied any metadata yet, the build process alerts you to the fact that it’s missing some important information like a
README file, the author, and so on. Adding this information is covered later in this chapter.
Notice that the build process installs the
wheel packages. Setuptools (https://setuptools.readthedocs.io) is a library that was, for a long time, one of the only ways to create Python packages. Now, Setuptools is one of a variety of available build backends for Python package builds.
DEFINITION: A build backend is a Python object that provides several required and optional hooks that implement packaging behavior. The core build backend interface is defined in PEP 517 (https://www.python.org/dev/peps/pep-0517/#build-backend-interface).
A build backend does the logistical work of creating package artifacts during the build process, namely through the
build_wheel hooks. Setuptools uses the
wheel package to build the wheel during the
build_wheel step. The
build tool uses Setuptools as a build backend by default when you don’t specify one.
The presence of build backends may leave you wondering if there may be build frontends as well. As it turns out, you’ve been using a build frontend already. The
build tool is a build frontend!
DEFINITION: A build frontend is a tool you run to initiate building a package from source code. The build frontend provides a user interface and integrates with the build backend via the hook interface.
To recap, you use a build frontend tool like
build to trigger a build backend like Setuptools to create package artifacts from your source code and metadata (figure 1).
Figure 1. The Python build system consists of a frontend user interface that integrates with a backend to build package artifacts.
Because the build process creates package artifacts, you can now check the effect of running the build. List the contents of the root directory for your package now. You should see the following:
$ ls -a1 $HOME/code/first-python-package/ . .. .venv/ UNKNOWN.egg-info/ build/ dist/ pyproject.toml
build/ directories are intermediate artifacts. List the contents of the
dist/ directory, where you should see the source and binary wheel package files:
$ ls -a1 $HOME/code/first-python-package/dist/ UNKNOWN-0.0.0-py3-none-any.whl UNKNOWN-0.0.0.tar.gz
Other build system tools
As I mentioned earlier, other options exist for both build frontends and backends. Some packages provide both a frontend and a backend, but I like
build and Setuptools.
If you want to explore some alternative build tools, check out poetry (https://python-poetry.org/) and flit (https://flit.readthedocs.io). Each build system makes different trade-offs between ease of configuration, capability, and user interface. As an example, flit and poetry are geared toward pure-Python packages whereas Setuptools can support extensions in other languages.
You can switch to another build system in a couple of steps:
- Install the new build frontend package
pyproject.tomlto specify the new build backend and its requirements
- Move the metadata about the package to the location expected by the new build backend
build used Setuptools as the fallback build backend because you didn’t specify one. You can specify Setuptools as the build backend for your package by adding the lines in listing 2 to
pyproject.toml. These lines specify the following:
build-system— This section describes the package build system.
requires— These are a list of dependencies, as strings, which must be installed for the build system to work. A Setuptools build system needs
wheelas you saw earlier in this chapter.
build-backend— This identifies the entrypoint to the build backend object, using the dotted path as a string. The Setuptools build backend object is available at
These represent the complete configuration you need to specify the build backend.
Listing 2. A build system backend specification to use Setuptools
[build-system] ❶ requires = ["setuptools", "wheel"] ❷ build-backend = "setuptools.build_meta" ❸
❶ Opens a new TOML section
❷ List of package names as strings
❸ Dotted path to the object as a string
Once you’ve added the build system information, run the build again. Nothing should change in the output: you’ve just locked in Setuptools as the explicit backend instead of letting
build fall back to it as a default. Now that you’ve got a handle on the Python package build system, you need to add some metadata about your package. You can read all about it in the book here.