The State Of Python ‘setuptools’ ‘pyproject.toml’ Packaging In 2024
I recently needed to package some python and read over a couple of old posts.
I pointed out some issues at the time and have found that they have been solved and managing packages is a little neater.
Python Package Example
We’re going to use the original pkgexample I created at the time. You can check out a copy here,
Let’s start by checking out the repo,
ubuntu@goodboy:~/myrepos/rex$ git clone [email protected]:RexBytes/pkgexample.git
Cloning into 'pkgexample'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 13 (delta 0), reused 10 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), done.
ubuntu@goodboy:~/myrepos/rex$ cd pkgexample
ubuntu@goodboy:~/myrepos/rex/pkgexample$ git setrbCode language: JavaScript (javascript)and just check that the build process works,
ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
LICENCE pyproject.toml README.md src
ubuntu@ubuntu-ThinkPad-X260:~/myrepos/rex/pkgexample$ python3 -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=65.0)
* Getting build dependencies for sdist...
.... some more output...
creating '/home/ubuntu/myrepos/rex/pkgexample/dist/.tmp-g16ah0km/pkgexample-1.0.2-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pkgexample/__init__.py'
adding 'pkgexample/helloworld.py'
adding 'pkgexample-1.0.2.dist-info/LICENCE'
adding 'pkgexample-1.0.2.dist-info/METADATA'
adding 'pkgexample-1.0.2.dist-info/WHEEL'
adding 'pkgexample-1.0.2.dist-info/top_level.txt'
adding 'pkgexample-1.0.2.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built pkgexample-1.0.2.tar.gz and pkgexample-1.0.2-py3-none-any.whl
ubuntu@goodboy:~/myrepos/rex/pkgexample$
ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
dist LICENCE pyproject.toml README.md src
ubuntu@goodboy:~/myrepos/rex/pkgexample$ cd dist/
ubuntu@goodboy:~/myrepos/rex/pkgexample/dist$ ls
pkgexample-1.0.2-py3-none-any.whl pkgexample-1.0.2.tar.gz
Code language: JavaScript (javascript)okay so that works. Notice the new dist directory. Remove that and return the checked out directory to it’s original checked out form.
Editable Install
An editable install of a python package allows you to install your package but you can still edit and develop your package code live in the directory you checked out the code to.
If you remember there were some issues doing this and we had to carry out a workaround if not using a venv. I won’t mention these workarounds here as it looks like we don’t need them!!
Here is the command you need to run from the root directory of your package to install it in editable mode.
python3 -m pip install -e .Would you look at this!!! It installed with no problems.
ubuntu@goodboy:~/myrepos/rex/pkgexample$ ls
LICENCE pyproject.toml README.md src
ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
Building editable for pkgexample (pyproject.toml) ... done
Created wheel for pkgexample: filename=pkgexample-1.0.2-0.editable-py3-none-any.whl size=3053 sha256=9dd030322a2477cfb942dec42e8cfce0143b178d74bb9ae4085ebb2a244855e9
Stored in directory: /tmp/pip-ephem-wheel-cache-ghnaac_9/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
DEPRECATION: pdfminer-six -VERSION- has a non-standard version number. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pdfminer-six or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.2
Code language: JavaScript (javascript)Is it actually installed? Let’s check it out.
ubuntu@goodboy:~$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld=HelloWorld()
>>> my_helloworld.print_message()
Hello World
>>> Code language: JavaScript (javascript)In 2024, it actually works. If you check the site-packages directory you will find a text file with the name of your package prefixed with the string ‘__editable__’, and if you check the contents you will find the full path to your package code src.
ubuntu@goodboy:~/.local/lib/python3.10/site-packages$ cat __editable__.pkgexample-1.0.2.pth
/home/ubuntu/myrepos/rex/pkgexample/src
Code language: JavaScript (javascript)Let’s uninstall the package and return the checked out repo to its origional form.
ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.2
Uninstalling pkgexample-1.0.2:
Would remove:
/home/ubuntu/.local/lib/python3.10/site-packages/__editable__.pkgexample-1.0.2.pth
/home/ubuntu/.local/lib/python3.10/site-packages/pkgexample-1.0.2.dist-info/*
Proceed (Y/n)? y
Successfully uninstalled pkgexample-1.0.2Code language: JavaScript (javascript)Editable Install With Venvs
Following the linked tutorials, we’re going to create a virtual environment and install the package in editable mode.
ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m venv ~/myvenv
ubuntu@goodboy:~/myrepos/rex/pkgexample$ source ~/myvenv/bin/activate
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
Building editable for pkgexample (pyproject.toml) ... done
Created wheel for pkgexample: filename=pkgexample-1.0.2-0.editable-py3-none-any.whl size=3053 sha256=bde4f97dc89b0620471a8d86733ecd99fbfd2ca21059b231c9413767dbca9688
Stored in directory: /tmp/pip-ephem-wheel-cache-6drshwf0/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.2
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ Code language: PHP (php)Well, it looks like the install of the package inside a venv with editable mode worked. Let see if we can call it.
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld=HelloWorld()
>>> my_helloworld.print_message()
Hello World
>>> Code language: JavaScript (javascript)It looks like we can! This worked!
I’m now curious, am going to have a peek at the site-package directory.
ubuntu@goodboy:~$ cd ~/myvenv
ubuntu@goodboy:~/myvenv$ ls
bin include lib lib64 pyvenv.cfg
ubuntu@goodboy:~/myvenv$ cd lib
ubuntu@goodboy:~/myvenv/lib$ ls
python3.10
ubuntu@goodboy:~/myvenv/lib$ cd python3.10/
ubuntu@goodboy:~/myvenv/lib/python3.10$ ls
site-packages
ubuntu@goodboy:~/myvenv/lib/python3.10$ cd site-packages/
ubuntu@goodboy:~/myvenv/lib/python3.10/site-packages$ ls
_distutils_hack __editable__.pkgexample-1.0.2.pth pip-22.0.2.dist-info pkg_resources setuptools-59.6.0.dist-info
distutils-precedence.pth pip pkgexample-1.0.2.dist-info setuptools
ubuntu@goodboy:~/myvenv/lib/python3.10/site-packages$ cat __editable__.pkgexample-1.0.2.pth
/home/ubuntu/myrepos/rex/pkgexample/srcCode language: PHP (php)Exactly what I expected. There is also a file with the name of the package prefixed with ‘__editable__‘.
Okay, let’s clean all of this up. Let’s uninstall the package, deactivate the venv, and then delete the venv.
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.2
Uninstalling pkgexample-1.0.2:
Would remove:
/home/ubuntu/myvenv/lib/python3.10/site-packages/__editable__.pkgexample-1.0.2.pth
/home/ubuntu/myvenv/lib/python3.10/site-packages/pkgexample-1.0.2.dist-info/*
Proceed (Y/n)? y
Successfully uninstalled pkgexample-1.0.2
(myvenv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ deactivate
ubuntu@goodboy:~/myrepos/rex/pkgexample$ rm -rf ~/myvenvCode language: JavaScript (javascript)Final Note:
Well, I am very impressed. You can now package purely with TOML, and install your package in editable mode without the messy workarounds we had to do last time. You can do this with, or without, a venv.
I may have been late to the party.
