Python Package Interactive Install / Editable Install With “pyproject.toml”

An interactive or editable python package install enables you to see real time changes when developing a package. Instead of uninstalling, repackaging, and reinstalling your package an interactive install immediately reflects your code changes.

Software Version Checks

Just for the record I’m going to check the version of all of the packages I have installed on my system, the packages needed in the build and install process.

ubuntu@goodboy:~$ python3 -m pip show setuptools | grep Version
Version: 65.3.0
ubuntu@goodboy:~$ python3 -m pip show build | grep Version
Version: 0.8.0
ubuntu@goodboy:~$ python3 -m pip show twine | grep Version
Version: 4.0.1
ubuntu@goodboy:~$ python3 -m pip show pip | grep Version
Version: 22.2.2
ubuntu@goodboy:~$ 

Let’s also note the Ubuntu version I am running on too.

ubuntu@goodboy:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

I’m doing this check as it seems I have caught Python Packaging with TOML at an interesting time. As of today, 31st August 2022, reading this documentation interactive installs (or editable installs) has just been merged in to main for the “setuptools” build system.

Interactive Install / Editable Install Attempt With Virtual Environment Venv

To follow along you will need a python package, if you don’t have your own please follow this post first to create your first python package using the “setup tools” build system. Come back here when ready.

I’m going to use the package “pkgexample” we created in that other post, and were going to try and install it in editable mode.

ubuntu@goodboy:~/myrepos/rex$ tree -a pkgexample/

pkgexample/
├── .gitignore
├── LICENCE
├── pyproject.toml
├── README.md
└── src
    └── pkgexample
        ├── helloworld.py
        └── __init__.py

2 directories, 6 files
ubuntu@goodboy:~/myrepos/rex$

From the root directory we’re going to run the following code.

python3 -m pip install -e .

Let’ see how this turns out for us.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
ERROR: Project file:///home/ubuntu/myrepos/rex/pkgexample has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Not very well.

ERROR: Project file:///home/ubuntu/myrepos/rex/pkgexample has a ‘pyproject.toml’ and its build backend is missing the ‘build_editable’ hook. Since it does not have a ‘setup.py’ nor a ‘setup.cfg’, it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

If this is how you remember editable installs the above error message probably surprised you.

Things change – come one, let’s take a look.

The “setuptools” documentation now recommends creating a virtual environment, and then running the interactive install command – lets try their recommended approach. I’m going to create my venv in my root directory, i’m unimaginatively calling it venv.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m venv ~/venv
ubuntu@goodboy:~/myrepos/rex/pkgexample$ source ~/venv/bin/activate
(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ pip install -e .
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
  Building editable for pkgexample (pyproject.toml) ... done
  Created wheel for pkgexample: filename=pkgexample-1.0.1-0.editable-py3-none-any.whl size=3060 sha256=9a01c34f780d08b90f688fa966d16d8c283727022dc7bb4cc63f625346e78f0a
  Stored in directory: /tmp/pip-ephem-wheel-cache-lyd0t8zf/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.1
(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$

Yea, it worked.

We have dropped in to a virtual environment ( more on these in another post), and it looks like the package has been installed interactively. Let’s test this.

Opening a Python3 shell, and importing our package modules does show it as installed.

(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld_object=HelloWorld()
>>> my_helloworld_object.print_message()
Hello World
>>>

Now exiting, exit(), the python shell, I am going to make a change to my helloworld.py module and change the message to “Hello Interactive World” and save the changes.

class HelloWorld:
    def __init__(self):
        self.a_simple_message="Hello Interactive World"

    def print_message(self):
        print(self.a_simple_message)

Opening a python shell and running the same commands should show a different result.

(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld_object=HelloWorld()
>>> my_helloworld_object.print_message()
Hello Interactive World
>>>

It does.

Interactive Install With No Virtual Env

Want to stick with the old ways? You might be able to with this work around.

Uninstall Your Package And Virtual Environment

If you want to attempt this workaround but played along with the above recommended venv approach, please apply the following instructions to uninstall the package “pkgexample”, deactivate and delete your virtual environment.

(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip uninstall pkgexample
Found existing installation: pkgexample 1.0.1
Uninstalling pkgexample-1.0.1:
  Would remove:
    /home/ubuntu/myrepos/rex/pkgexample/.venv/lib/python3.10/site-packages/__editable__.pkgexample-1.0.1.pth
    /home/ubuntu/myrepos/rex/pkgexample/.venv/lib/python3.10/site-packages/pkgexample-1.0.1.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled pkgexample-1.0.1
(.venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$

Then deactivate your virtual environment / venv.

(venv) ubuntu@goodboy:~/myrepos/rex/pkgexample$ deactivate
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Then delete your venv directory.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ rm -rf ~/venv/
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Your project directory should now be restored to its original state, and the package “pkgexample” should be now uninstalled.

The Work Around

This is a little bit hacky, and I don’t think it is recommended procedure.

But here is the process.

If you re-read the error message we saw earlier and focus in on this part, you will see it wants a ‘setup.py’ file.

Since it does not have a ‘setup.py’ nor a ‘setup.cfg’

The current “setuptools” documentation doesn’t cover trying to do what the error message says so I am not sure if they intend this approach to be available longer term. Lets go ahead and do it anyway.

Give the error what it wants and create a new python module at the root of your project, name it ‘setup.py’.

ubuntu@goodboy:~/myrepos/rex$ tree -a pkgexample/

pkgexample/
├── .gitignore
├── LICENCE
├── pyproject.toml
├── README.md
├── setup.py              <----New File
└── src
    └── pkgexample
        ├── helloworld.py
        └── __init__.py

2 directories, 6 files
ubuntu@goodboy:~/myrepos/rex$

Add the following contents. Save and exit.

from setuptools import setup

if __name__ == '__main__':
    setup()

Now you would expect running the interactive install command to now work.
Unfortunately it errors out.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
    error: subprocess-exited-with-error

Apparently this is a bug as described here. It is marked as closed, but clearly it appears not to have been fixed.

Oh well… there is a work around for this bug and that is to use the –no-build-isolation switch. The modified interactive install command is now as follows.

python3 -m pip install --no-build-isolation -e .

More details on the –no-build-isolation switch can be found here.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3 -m pip install --no-build-isolation -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/myrepos/rex/pkgexample
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pkgexample
  Building editable for pkgexample (pyproject.toml) ... done
  Created wheel for pkgexample: filename=pkgexample-1.0.1-0.editable-py3-none-any.whl size=3060 sha256=a65f9d3289ab1a92cb82bb8148b3bd20c1fb8727b682f2f68868a0200e2406a5
  Stored in directory: /tmp/pip-ephem-wheel-cache-vcmukrup/wheels/12/47/38/a45fdd3c712bbe4ff26cfdbab7c85e633230070d7f3f95f9b1
Successfully built pkgexample
Installing collected packages: pkgexample
Successfully installed pkgexample-1.0.1
ubuntu@goodboy:~/myrepos/rex/pkgexample$

Does this workaround work? Yes it does. Here is the output when changing my message value with this workaround interactive install.

ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld_object=HelloWorld()
>>> my_helloworld_object.print_message()
Hello World
>>> exit()
ubuntu@goodboy:~/myrepos/rex/pkgexample$ python3
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgexample.helloworld import HelloWorld
>>> my_helloworld_object=HelloWorld()
>>> my_helloworld_object.print_message()
Hello Workaround World
>>>

Closing Notes

Well, as you can see, the current state of interactive / editable installs with “pyproject.toml” configuration files and the “src/source layout” package structure is a little rough around the edges.

Not too rough, and does work with some minimal workarounds.
When they get this fixed for people who don’t want to use virtual environments, it will be awesome.
Using virtual environments do have some big advantages though.

Stay tuned for further updates in the exciting world of interactive installs.

Let me know in the comments if you find out anything extra.

Further Reading

https://github.com/pypa/setuptools/pull/3488

https://github.com/pypa/setuptools/pull/3488

https://github.com/pypa/setuptools/blob/main/docs/userguide/development_mode.rst