Python Packaging – Using “setuptools”, “pyproject.toml”

Register An Account At pypi.org

If you haven’t done so already, head over to pypi.org and register an account as soon as you can. Pypi.org is the official third-party software repository for python.

We will have you creating, uploading and sharing your packages through pypi.org in no time.

Package Up Your Code

A Simple HelloWorld Class

To demonstrate packaging I’m going to need a some python code, here I have created a simple python project consisting of just one HelloWorld class. Your projects may be a bit more complex and contain more that just the one class.

class HelloWorld:
    def __init__(self):
        self.a_simple_message="Hello World"

    def print_message(self):
        print(self.a_simple_message)

The SRC/Source Package Layout

I’m going to wrap up the helloworld.py class file inside a python packaging structure known as the “src/source layout”.

More information on the package source layout from python documentation here.

ubuntu@goodboy:~/myrepos/rex$ tree -a pkgexample/

pkgexample/
├── .gitignore
├── LICENCE
├── pyproject.toml
├── README.md
└── src
    └── pkgexample
        ├── helloworld.py
        └── __init__.py

2 directories, 6 files
ubuntu@goodboy:~/myrepos/rex$

The SRC Subdirectory

It is named the “source layout” as any package you want to distribute is placed under the inner src/ subdirectory.

Since I have decided on the package name “pkgexample” to contain our simple python project you will see a directory of that name in the inner src/ directory.

The SRC Package Root Directory Name

The root directory of the “src/source layout” package structure can be named absolutely anything, go ahead and name it “bananas” if you wish – but it is more convenient for everyone if you give it the same name as the package you want to distribute.

I have named the root directory “pkgexample” to match the package found at “src/pkgexample”.

Python Package And Module Definitions

Regular Python Package

The definition of a regular python package is any directory that contains an “__init__.py” file. This file is a marker to let python know it can import code modules from that directory.

The “__init__.py” file is usually empty, but as your projects grow you can make entries inside the “__init__.py” file to export parts of your complex package with more convenient names.

If there is no “__init__.py” file present, python will skip over a directory and you will not be able to import your package and modules.

ubuntu@goodboy:~/myrepos/rex$ tree -a pkgexample/

pkgexample/
├── .gitignore
├── LICENCE
├── pyproject.toml
├── README.md
└── src
    └── pkgexample
        ├── helloworld.py
        └── __init__.py

2 directories, 6 files
ubuntu@goodboy:~/myrepos/rex$

Python Modules

A “python module” is just a name that refers to any file that contains python code.

Our simple helloworld.py python file is a python module and is contained in our python package directory pkgexample/ , which is contained within the src/ directory.

Package Documentation Files “README.md”, “LICENCE”

Tell me you provide high quality documentation for you code. You do document don’t you?

Inside the first level of the root directory you should have the files README.md and LICENCE.

README.md

The README.md file contains all of your great documentation.

It is usually written in markdown. More information on markdown here, or the GitHub flavored markdown here.

LICENCE

Look in to licensing for your specific needs. GitHub does provide license templates for your project repositories. I usually go for the MIT license.

Project Configuration File “pyproject.toml”

This is a configuration file written in TOML, more information on TOML here.

This is the most important file, apart from your project code, in the packaging process. It tells the python packager what build system to use, information about your project, and where to look for further information.

The “pyproject.toml” configuration file has the following key/value pair tables: [build-system], [project] and [project.urls] .

[build-system]
requires = ["setuptools>=65.0"]
build-backend = "setuptools.build_meta"

[project]
name = "pkgexample"
version = "1.0.2"
authors = [
  { name="Good Boy", email="[email protected]" },
]
description = "A simple package example"
readme = "README.md"
license = { file="LICENSE" }
requires-python = ">=3.7"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]


[project.urls]
"Homepage" = "https://github.com/RexBytes/pkgexample"
"Bug Tracker" = "https://github.com/RexBytes/pkgexample/issues"
"Download URL" ="https://github.com/RexBytes/pkgexample/archive/refs/tags/v1.0.2.tar.gz"

[build-system]

This is where you state the build system you want to use for your python packaging. I’m going to use setup tools, and have required version greater than or equal to 65.

Other build systems are available such as, Hatching, Flit and PDM.

[project ]

In this table you provide information about your python project. At the bare minimum you should include the above. Most of the values are self explanatory, such as “name”, “version”, “authors”, “description”, “readme”, and “license”. Other fields need a little explaining.

Field “classifiers”

These field classifiers are used to help people find your project when searching on pypi.org. A full list of classifiers you can copy and past from can be found here.

Field “required-python”

If you have build your project to target specific python versions, it is here where you state the compatible versions of python for your project.

[project.url]

Here you can place any, and all URLs you want for your project.

For full details on these fields and others, look here.

The “.gitignore” File

Although this isn’t part of the python packaging process, and is not a required file, it is very useful to include.

When building python packages a few intermediary files are created, egg files, wheel files and tar files. These are your pypi.org package files and shouldn’t really be committed to your GitHub repository – it will get messy and disorganized fast.

*.egg-info
*.whl
*.tar.gz

At the very least you should have the above entries in your “.gitignore” file.
There are many general “.gitignore” templates available out there, here is one designed to cover almost any python project you create.

GitHub Repository

For convenience the package “pkgexample” and its files are available at the following GitHub repository.

Building And Uploading Your Distribution Archives

Now that you have a valid package directory structure containing your python package and modules, the fun starts.

Software Requirements

Let’s get these software requirements out of the way quick, these are all needed to package, check and upload your package to pypi.org

Python3 pip

We need the latest version of pip installed for python 3.
Run the following command to install and upgrade your pip.

python3 -m pip install --upgrade pip

PyPA build

We need the latest PyPA’s build installed to build your package.

python3 -m pip install --upgrade build

Twine Package Uploader

Twine is used to check, and upload your package to your pypi.org account.

python3 -m pip install --upgrade twine

Building Your Package

This part is going to be really short. We’re just going to run the packaging commands, and your package will soon be ready to be installed by your mates.

Here, from the root directory, we run the build command and as you can see, it finished successfully.

You will find that a dist/ directory magically appears in your root directory, this is where your distribution has been packaged to.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=61.0)
* Getting dependencies for sdist...
...
...
...
Successfully built pkgexample-1.0.2.tar.gz and pkgexample-1.0.2-py3-none-any.whl

Check Your Distribution Before Uploading

Twine only checks that your package is packaged correctly – it doesn’t test your code, that is entirely your responsibility.

Again from the root directory, ask twine to check your newly compiled distribution inside your ./dist/ directory.

Our simple package “pkgexample” has passed all the tests and is ready to be published.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m twine check ./dist/*
Checking ./dist/pkgexample-1.0.2-py3-none-any.whl: PASSED
Checking ./dist/pkgexample-1.0.2.tar.gz: PASSED

Upload And Publish Your Package

LEGACY: Using username and password

pypi.org has warned that it will soon stop support for username/password upload, keep that in mind and read the next section for the API Key method.

Finally, again from your root directory, use the following command to upload and publish your python package.

You will be asked for your pypi.org username and password.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m twine upload ./dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: rexbytes
Enter your password:
Uploading pkgexample-1.0.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 kB • 00:00 • 82.3 kB/s
Uploading pkgexample-1.0.2.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 kB • 00:00 • ?
View at:
https://pypi.org/project/pkgexample/1.0.2/
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Upload Using __token__ and API key

Full documentation from pypi.org can be found here, but the following will get you up and running.

You must first generate an API Token under your pypi.org account, click through account settings and scroll to the bottom where you will find an [Add API Token] button. Generate your Token and keep it safe.

Uploading your package is the exact same process except when providing credentials, when asked for your username use “__token__” note the double underscore on each side. The password is your API token, it’s quite long so copy paste it then press enter. Your password won’t be displayed.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m twine upload ./dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: __token__
Enter your password:
Uploading pkgexample-1.0.2-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 kB • 00:00 • 82.3 kB/s
Uploading pkgexample-1.0.2.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 kB • 00:00 • ?
View at:
https://pypi.org/project/pkgexample/1.0.2/
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Enjoy Your Freshly Uploaded Project

Head over to https://pypi.org/project/pkgexample/ where you should see the “pkgexample” package live and ready to be used in other people’s projects. (Of course your package location for your project will be in a different place)

Using Your Package

You should try all of the following commands, “pkgexample” is now a real package hosted on PyPi.org

Open up a fresh ubuntu terminal to install the “pkgexample” package.

ubuntu@goodboy:~$ python3 -m pip install pkgexample
Defaulting to user installation because normal site-packages is not writeable
Collecting pkgexample
  Downloading pkgexample-1.0.2-py3-none-any.whl (3.3 kB)
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.2

It is now installed on your system.

Let’s test it. Open up an interactive Python3 shell and play along with the following code. It should work as expected.

ubuntu@goodboy:~$ python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld_object=HelloWorld()
>>> my_helloworld_object.print_message()
Hello World
>>>

Exit, exit(), your python shell and let’s uninstall the package for now.

ubuntu@goodboy:~$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.2
Uninstalling pkgexample-1.0.2:
  Would remove:
    /home/ubuntu/.local/lib/python3.10/site-packages/pkgexample-1.0.2.dist-info/*
    /home/ubuntu/.local/lib/python3.10/site-packages/pkgexample/*
Proceed (Y/n)? y
  Successfully uninstalled pkgexample-1.0.2
ubuntu@goodboy:~$

Keep it installed if you wish.

Python Packages Installed As UNKNOWN

If you stray from the software requirements at the beginning you might get the ‘Installing collected packages: UNKNOWN‘ issue, which is annoying.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install --no-build-isolation -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Checking if build backend supports build_editable ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
Successfully installed UNKNOWN-0.0.0

You can either upgrade setuptools to the latest version to see if they have fixed the issue,

ubuntu@goodboy:~/myrepos/rex/pkgexample$ pip3 install --upgrade setuptools
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
Collecting setuptools
  Using cached setuptools-65.4.1-py3-none-any.whl (1.2 MB)
Installing collected packages: setuptools
Successfully installed setuptools-65.4.1

or you can install a version of setuptools that you know works for you, here is the command to install the version stated at the beginning of this article,

ubuntu@goodboy:~/myrepos/rex/pkgexample$ pip3 install setuptools==65.3.0
Defaulting to user installation because normal site-packages is not writeable
Collecting setuptools==65.3.0
  Downloading setuptools-65.3.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.2 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-65.3.0

Further Reading

https://packaging.python.org/en/latest/tutorials/packaging-projects/

https://setuptools.pypa.io/en/latest/

https://hatch.pypa.io/latest/

https://flit.pypa.io/en/latest/

https://pdm.fming.dev/latest/

https://toml.io/en/

https://www.markdownguide.org/getting-started/

https://github.github.com/gfm/

https://linuxhint.com/bash-tree-command/