Packaging applications to install on other machines with Python
$ rpmqa_simple.py --limit 20 linux-firmware-20210818: 395,099,476 code-1.61.2: 303,882,220 brave-browser-1.31.87: 293,857,731 libreoffice-core-220.127.116.11: 287,370,064 thunderbird-91.1.0: 271,239,962 firefox-92.0: 266,349,777 glibc-all-langpacks-2.32: 227,552,812 mysql-workbench-community-8.0.23: 190,641,403 java-11-openjdk-headless-18.104.22.168.8: 179,469,639 iwl7260-firmware-22.214.171.124: 148,167,043 docker-ce-cli-20.10.10: 145,890,250 google-noto-sans-cjk-ttc-fonts-20190416: 136,611,853 containerd.io-1.4.11: 113,368,911 ansible-2.9.25: 101,106,247 docker-ce-20.10.10: 100,134,880 ibus-1.5.23: 90,840,441 llvm-libs-11.0.0: 87,593,600 gcc-10.3.1: 84,899,923 cldr-emoji-annotation-38: 80,832,870 kernel-core-5.14.12: 79,447,964
Now I want to package an application so that I can install it easily, including all the dependencies, on other machines. In this article, I'll show how to use the setuptools package to do that.
That's a lot to cover, so basic knowledge of Python is required. Even if you don't know much, the code is simple to follow, and the boilerplate code is small.
Set up your environment
My previous article explained how to install the code and use virtual environments, but here is a shortcut:
$ sudo dnf install -y python3-rpm $ git clone firstname.lastname@example.org:josevnz/tutorials.git $ cd rpm_query $ python3 -m venv --system-site-packages ~/virtualenv/rpm_query . ~/virtualenv/rpm_query/bin/activate (rpm_query)$
Package and install the distribution
The next step is to package the application. However, I won't use RPM for that.
Why not use an RPM to package the Python application?
Well, there is no short answer to that.
RPM is great if you want to share your application with all the users of your system, especially because RPM can automatically install related dependencies.
For example, the RPM Python bindings (
rpm-python3) are distributed this way, which makes sense as it is thinly tied to RPM.
In some cases, it is also a disadvantage:
- Users need root elevated access to install an RPM. If the code is malicious, it will take control of the server very easily. (That's why you always check the RPM signatures and download code from well-known sources, right?)
- Attempting to upgrade to an RPM that is incompatible with older dependent applications fails.
- RPM is not well suited to share "test" code created during continuous integration, at least in bare-metal deployments. With a container, that is probably a different story.
- If the Python code has dependencies, they will likely have to be packaged as RPMs.
Use virtual environments and pip + setuptools
Using virtual environments, pip, and setup tools solves these RPM limitations because:
- Virtual environments allow installing applications without elevated permissions.
- The application is self-contained in the virtual environment. Administrators can install different versions of the libraries without affecting the whole system.
- It is very easy to integrate a virtual environment with continuous integration and unit testing. After the tests pass, the environment can be recycled.
- setuptools solves the problem of packaging your application in a nice directory structure for making scripts and libraries available to users.
- setuptools also deals with keeping track of dependencies by using proper version checks to make the build process repeatable.
- setuptools works with pip, the Python package manager.
- Both virtual environments and setuptools have excellent support in IDEs like PyCharm and VS Code.
[ Decrease the complexity of getting into the cloud by downloading the Hybrid cloud strategy for dummies eBook. ]
Work with setuptools
Once you're ready to deploy the application, you can package it, copy its wheel file, and then install it in a new virtual environment. First, define a very important file,
setup.py, which is used by setuptools.
The most important sections in the file below are:
- *_requires sections: Build and installation dependencies
- packages: The location of Python classes
- scripts: The scripts that the end user calls to interact with the libraries (if any)
""" Project packaging and deployment More details: https://setuptools.pypa.io/en/latest/userguide/quickstart.html """ import os from setuptools import setup from reporter import __version__ def __read__(file_name): return open(os.path.join(os.path.dirname(__file__), file_name)).read() setup( name="rpm_query", version=__version__, author="Jose Vicente Nunez Zuleta", author_email="email@example.com", description=__doc__, license="Apache", keywords="rpm query", url="https://github.com/josevnz/rpm_query", packages=[ 'reporter' ], long_description=__read__('README.md'), # https://pypi.org/pypi?%3Aaction=list_classifiers classifiers=[ "Development Status :: 3 - Alpha", "Topic :: Utilities", "Environment :: X11 Applications", "Intended Audience :: System Administrators" "License :: OSI Approved :: Apache Software License" ], setup_requires=[ "setuptools==49.1.3", "wheel==0.37.0", "rich==9.5.1", "dearpygui==1.0.2" ], install_requires=[ "rich==9.5.1", ], scripts=[ "bin/rpmq_simple.py", ] )
Things to note:
- I stored the version in the
reporter/__init__.pypackage module to share with other parts of the application, not just setuptools. I also used a semantic version schema naming convention.
- The module's README is also stored as an external file. This makes editing the file much easier without worrying about size or breaking the Python syntax.
- Classifiers make it easier to see the application's intent.
- I can define packaging dependencies (
setup_requires) and runtime dependencies (
- I need the
wheelpackage to create a precompiled distribution that is faster to install than other modes.
pyproject.toml file is also very important. Here is the file:
[build-system] requires = ["setuptools", "wheel"]
pyproject.toml file specifies what is used for packaging the scripts and installing from source.
Quick check before uploading
Before you upload the wheel, you should ask Twine to check your settings for errors, like this:
(rpm_query) [josevnz@dmaf5 rpm_query]$ twine check dist/rpm_query-0.0.1-py3-none-any.whl Checking dist/rpm_query-0.0.1-py3-none-any.whl: FAILED `long_description` has syntax errors in markup and would not be rendered on PyPI. line 20: Warning: Bullet list ends without a blank line; unexpected unindent. warning: `long_description_content_type` missing. defaulting to `text/x-rst`.
The Markdown is incorrect in the file. One way to fix this issue is to install:
$ pip install readme_renderer[md]
long_description_content_type section is there:
If you rerun it after making the changes above, you still see the warning:
rpm_query) [josevnz@dmaf5 rpm_query]$ twine check dist/rpm_query-0.0.1-py3-none-any.whl Checking dist/rpm_query-0.0.1-py3-none-any.whl: PASSED, with warnings warning: `long_description_content_type` missing. defaulting to `text/x-rst`.
No serious errors and one false alarm. You are ready to upload your wheel.
How to deploy while testing
You don't need to package and deploy the application in full mode. This is because setuptools has a very convenient develop mode that installs dependencies and allows editing the code while testing it:
(rpm_query)$ python setup.py develop
This mode creates special symbolic links that put the scripts (remember that
scripts: section inside
setup.py?) into your path.
By the way, once you are finished testing, you can remove development mode:
(rpm_query)$ python setup.py develop --uninstall
The official documentation recommends migrating from a
setup.py configuration to
setup.cfg. I decided to use
setup.py because it is still the most popular format.
Create a precompiled distribution
Compiling is as simple as typing this:
(rpm_query)$ python setup.py bdist_wheel running bdist_wheel ... # Omitted output (rpm_query)$ ls dist/ rpm_query-0.0.1-py3-none-any.whl
Then you can install it on the same machine or a new virtual machine:
(rpm_query)$ python setup.py install \ dist/rpm_query-1.0.0-py3-none-any.whl
Or with the new preferred way, using
build. First make sure the module is installed:
(rpm_query) $ pip install build Collecting build Downloading build-0.7.0-py3-none-any.whl (16 kB) Collecting tomli>=1.0.0 Downloading tomli-1.2.2-py3-none-any.whl (12 kB) Requirement already satisfied: packaging>=19.0 in /usr/lib/python3.9/site-packages (from build) (20.4) Collecting pep517>=0.9.1 Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB) Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3.9/site-packages (from packaging>=19.0->build) (2.4.7) Requirement already satisfied: six in /usr/lib/python3.9/site-packages (from packaging>=19.0->build) (1.15.0) Installing collected packages: tomli, pep517, build Successfully installed build-0.7.0 pep517-0.12.0 tomli-1.2.2
And then you can package your module like this (note that we do not tell
build to use a virtual environment because we are already in one):
(rpm_query) $ python3 -m build --no-isolation * Getting dependencies for wheel... * Building wheel... running bdist_wheel running build running build_scripts # ... Omitted output whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'rpm_query-0.1.0.data/scripts/rpmq_dearpygui.py' adding 'rpm_query-0.1.0.data/scripts/rpmq_rich.py' adding 'rpm_query-0.1.0.data/scripts/rpmq_simple.py' adding 'rpm_query-0.1.0.data/scripts/rpmq_tkinter.py' adding 'rpm_query-0.1.0.dist-info/LICENSE.txt' adding 'rpm_query-0.1.0.dist-info/METADATA' adding 'rpm_query-0.1.0.dist-info/WHEEL' adding 'rpm_query-0.1.0.dist-info/top_level.txt' adding 'rpm_query-0.1.0.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel Successfully built rpm_query-0.1.0-py3-none-any.whl
What if you want to share the .pip file with other users? I could copy the wheel file to the different machines and have the users install it, but there is a better way.
Set up a private PyPI server
Note: This setup is not production quality because:
- It is not secure because it uses passwords instead of tokens for authentication.
- There's no SSL encryption. HTTP sends clear-text passwords over the wire.
- There's no storage redundancy. Ideally, the pip storage should have some sort of redundancy and backups.
[ Download the datasheet for information about Red Hat OpenShift Container Platform. ]
First, create a directory to store the packages:
$ mkdir -p -v $HOME/pypiserver $ mkdir: created directory '/home/josevnz/pypiserver'
Then use htpasswd to set up a user and password to upload the packages:
htpasswd -c $HOME/.htpasswd josevnz New password: Re-type new password: Adding password for user josevnz
After that, run the container in a detached mode:
$ docker run --detach --name privatepypiserver --publish 8080:8080 --volume ~/.htpasswd:/data/.htpasswd --volume $HOME/pypiserver:/data/packages pypiserver/pypiserver:latest -P .htpasswd --overwrite packages f95f59a882b639db4509081de19a670fa8fdd93c63c3d4562c89e49e70bf6ee5 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f95f59a882b6 pypiserver/pypiserver:latest "/entrypoint.sh -P .…" 7 seconds ago Up 6 seconds 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp privatepypiserver
Confirm it is running by pointing curl or lynx to the new
[josevnz@dmaf5 ~]$ lynx http://localhost:8080 Welcome to pypiserver! Welcome to pypiserver! This is a PyPI compatible package index serving 0 packages. To use this server with pip, run the following command: pip install --index-url http://localhost:8080/simple/ PACKAGE [PACKAGE2...] To use this server with easy_install, run the following command: easy_install --index-url http://localhost:8080/simple/ PACKAGE [PACKAGE2...] The complete list of all packages can be found here or via the simple index. This instance is running version 1.4.2 of the pypiserver software.
Next, upload the wheel to your private PyPI server.
Upload the application to a repository with Twine
The most common way to share Python code is to upload it to an artifact manager like Sonatype Nexus or pypiserver using a tool like Twine.
(rpm_query)$ pip install twine
The next step is to set up ~/.pypirc to allow passwordless uploads to the local PyPI server:
[distutils] index-servers = pypi privatepypi [pypi] repository = https://upload.pypi.org/legacy/ [privatepypi] repository = http://localhost:8080/ username = josevnz
It's best to not put the
password = XXXX inside the file. Let Twine ask for it instead. Also, make the configuration accessible only to the owner:
$ chmod 600 ~/.pypirc
Finally, upload the wheel using the
(rpm_query) twine upload -r privatepypi dist/rpm_query-0.0.1-py3-none-any.whl Uploading distributions to http://localhost:8080/ Uploading rpm_query-0.0.1-py3-none-any.whl 100%|██████████████████████████████████
Confirm it was installed with
Index of packages rpm_query-0.0.1-py3-none-any.whl Commands: Use arrow keys to move, '?' for help, 'q' to quit, '<-' to go back. Arrow keys: Up and Down to move. Right to follow a link; Left to go back. H)elp O)ptions P)rint G)o M)ain screen Q)uit /=search [delete]=history list
Install from the local privatepypi server
Wait, don't leave yet. It is time to install the package from your private PyPI server:
First, tell pip to look for packages on the private PyPI server:
$ mkdir --verbose --parents ~/.pip $ cat<<PIPCONF>~/.pip/.pip.conf [global] extra-index-url = http://localhost:8080/simple/ trusted-host = http://localhost:8080/simple/ PIPCONF
To prove this works well, install it to a different virtual environment (or you can override any previous installation with
pip install --force):
$ python3 -m venv ~/virtualenv/test2 $ . ~/virtualenv/test2/bin/activate (test2)$ pip install --index-url http://localhost:8080/simple/ rpm_query Looking in indexes: http://localhost:8080/simple/ Collecting rpm_query Downloading http://localhost:8080/packages/rpm_query-0.0.1-py3-none-any.whl (12 kB) Collecting rich==9.5.1 Using cached rich-9.5.1-py3-none-any.whl (180 kB) Collecting pygments<3.0.0,>=2.6.0 Downloading Pygments-2.10.0-py3-none-any.whl (1.0 MB) |████████████████████████████████| 1.0 MB 5.4 MB/s Collecting colorama<0.5.0,>=0.4.0 Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB) Collecting typing-extensions<4.0.0,>=3.7.4 Downloading typing_extensions-126.96.36.199-py3-none-any.whl (26 kB) Collecting commonmark<0.10.0,>=0.9.0 Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB) |████████████████████████████████| 51 kB 5.8 MB/s Installing collected packages: typing-extensions, pygments, commonmark, colorama, rich, rpm-query Successfully installed colorama-0.4.4 commonmark-0.9.1 pygments-2.10.0 rich-9.5.1 rpm-query-0.0.1 typing-extensions-188.8.131.52
What you've learned
I provided a lot of information here. Here's what I covered:
- Package the application with setuptools.
- Run a private PyPI server using a container.
- Upload the generated wheel package to the private repository using Twine.
- Install the wheel from the private repository instead of a file using pip.
In my next article in this series, I'll explore writing user interface applications in Python.