The State Of Python ‘setuptools’ ‘pyproject.toml’ Packaging In 2024

I recently needed to package some python and read over a couple of old posts.

I pointed out some issues at the time and have found that they have been solved and managing packages is a little neater.

Python Package Example

We’re going to use the original pkgexample I created at the time. You can check out a copy here,

Let’s start by checking out the repo,

ubuntu@goodboy:~/myrepos/rex$ git clone [email protected]:RexBytes/pkgexample.git
Cloning into 'pkgexample'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 13 (delta 0), reused 10 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), done.
ubuntu@goodboy:~/myrepos/rex$ cd pkgexample
ubuntu@goodboy:~/myrepos/rex/pkgexample$ git setrb

and just check that the build process works,

ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
LICENCE  pyproject.toml  README.md  src
ubuntu@ubuntu-ThinkPad-X260:~/myrepos/rex/pkgexample$ python3 -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=65.0)
* Getting build dependencies for sdist...

.... some more output...

creating '/home/ubuntu/myrepos/rex/pkgexample/dist/.tmp-g16ah0km/pkgexample-1.0.2-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pkgexample/__init__.py'
adding 'pkgexample/helloworld.py'
adding 'pkgexample-1.0.2.dist-info/LICENCE'
adding 'pkgexample-1.0.2.dist-info/METADATA'
adding 'pkgexample-1.0.2.dist-info/WHEEL'
adding 'pkgexample-1.0.2.dist-info/top_level.txt'
adding 'pkgexample-1.0.2.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built pkgexample-1.0.2.tar.gz and pkgexample-1.0.2-py3-none-any.whl
ubuntu@goodboy:~/myrepos/rex/pkgexample$ 
ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
dist  LICENCE  pyproject.toml  README.md  src
ubuntu@goodboy:~/myrepos/rex/pkgexample$ cd dist/
ubuntu@goodboy:~/myrepos/rex/pkgexample/dist$ ls
pkgexample-1.0.2-py3-none-any.whl  pkgexample-1.0.2.tar.gz

okay so that works. Notice the new dist directory. Remove that and return the checked out directory to it’s original checked out form.

Editable Install

An editable install of a python package allows you to install your package but you can still edit and develop your package code live in the directory you checked out the code to.

If you remember there were some issues doing this and we had to carry out a workaround if not using a venv. I won’t mention these workarounds here as it looks like we don’t need them!!

Here is the command you need to run from the root directory of your package to install it in editable mode.

python3 -m pip install -e .

Would you look at this!!! It installed with no problems.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
LICENCE  pyproject.toml  README.md  src
ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
  Building editable for pkgexample (pyproject.toml) ... done
  Created wheel for pkgexample: filename=pkgexample-1.0.2-0.editable-py3-none-any.whl size=3053 sha256=9dd030322a2477cfb942dec42e8cfce0143b178d74bb9ae4085ebb2a244855e9
  Stored in directory: /tmp/pip-ephem-wheel-cache-ghnaac_9/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
DEPRECATION: pdfminer-six -VERSION- has a non-standard version number. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pdfminer-six or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.2

Is it actually installed? Let’s check it out.

ubuntu@goodboy:~$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld=HelloWorld()
>>> my_helloworld.print_message()
Hello World
>>> 

In 2024, it actually works. If you check the site-packages directory you will find a text file with the name of your package prefixed with the string ‘__editable__’, and if you check the contents you will find the full path to your package code src.

ubuntu@goodboy:~/.local/lib/python3.10/site-packages$ cat __editable__.pkgexample-1.0.2.pth 
/home/ubuntu/myrepos/rex/pkgexample/src

Let’s uninstall the package and return the checked out repo to its origional form.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.2
Uninstalling pkgexample-1.0.2:
  Would remove:
    /home/ubuntu/.local/lib/python3.10/site-packages/__editable__.pkgexample-1.0.2.pth
    /home/ubuntu/.local/lib/python3.10/site-packages/pkgexample-1.0.2.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled pkgexample-1.0.2

Editable Install With Venvs

Following the linked tutorials, we’re going to create a virtual environment and install the package in editable mode.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m venv ~/myvenv
ubuntu@goodboy:~/myrepos/rex/pkgexample$ source ~/myvenv/bin/activate
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
  Building editable for pkgexample (pyproject.toml) ... done
  Created wheel for pkgexample: filename=pkgexample-1.0.2-0.editable-py3-none-any.whl size=3053 sha256=bde4f97dc89b0620471a8d86733ecd99fbfd2ca21059b231c9413767dbca9688
  Stored in directory: /tmp/pip-ephem-wheel-cache-6drshwf0/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.2
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ 

Well, it looks like the install of the package inside a venv with editable mode worked. Let see if we can call it.

(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld=HelloWorld()
>>> my_helloworld.print_message()
Hello World
>>> 

It looks like we can! This worked!

I’m now curious, am going to have a peek at the site-package directory.

ubuntu@goodboy:~$ cd ~/myvenv
ubuntu@goodboy:~/myvenv$ ls
bin  include  lib  lib64  pyvenv.cfg
ubuntu@goodboy:~/myvenv$ cd lib
ubuntu@goodboy:~/myvenv/lib$ ls
python3.10
ubuntu@goodboy:~/myvenv/lib$ cd python3.10/
ubuntu@goodboy:~/myvenv/lib/python3.10$ ls
site-packages
ubuntu@goodboy:~/myvenv/lib/python3.10$ cd site-packages/
ubuntu@goodboy:~/myvenv/lib/python3.10/site-packages$ ls
_distutils_hack           __editable__.pkgexample-1.0.2.pth  pip-22.0.2.dist-info        pkg_resources  setuptools-59.6.0.dist-info
distutils-precedence.pth  pip                                pkgexample-1.0.2.dist-info  setuptools
ubuntu@goodboy:~/myvenv/lib/python3.10/site-packages$ cat __editable__.pkgexample-1.0.2.pth 
/home/ubuntu/myrepos/rex/pkgexample/src

Exactly what I expected. There is also a file with the name of the package prefixed with ‘__editable__‘.

Okay, let’s clean all of this up. Let’s uninstall the package, deactivate the venv, and then delete the venv.

(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.2
Uninstalling pkgexample-1.0.2:
  Would remove:
    /home/ubuntu/myvenv/lib/python3.10/site-packages/__editable__.pkgexample-1.0.2.pth
    /home/ubuntu/myvenv/lib/python3.10/site-packages/pkgexample-1.0.2.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled pkgexample-1.0.2
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ deactivate
ubuntu@goodboy:~/myrepos/rex/pkgexample$ rm -rf ~/myvenv

Final Note:

Well, I am very impressed. You can now package purely with TOML, and install your package in editable mode without the messy workarounds we had to do last time. You can do this with, or without, a venv.

I may have been late to the party.