9. Packaging python
9.1. Motivation and plan
We talk a lot about packaging, and there is much to be said. [FIXME: outline here the issues surrounding system-native packaging, versus handing in a single archive, versus containerized approaches, …]
Plan: to explore the myriad ways in which one can package python, and try to make sense of it. The ultimate goal is to find a clean way to make an rpm (or dpkg) out of a well-structured python package.
9.2. Prerequisites
An RPM-based (and later debian-based) system, or maybe VM or container.
9.3. Concepts and the big problem
The challenge is to deliver python software that might include C/C++/FORTRAN shared libraries, and to deliver it robustly.
This is made very difficult by the fact that Python has had many different packaging and delivery approaches over time.
Compare it to the old days of C, then the progression of ruby, golang, rust, …
9.4. Listing all the python packaging approaches
As with the distinction of dpkg and apt, and of rpm and yum/dnf, there is probably a similar distinction between a python package and a channel of packages with coordinated dependencies – maybe wheel vs. pip?
As I start writing (2021-10-26) I do not yet know enough to distinguish these, especially because I think that wheels are superseded by new approaches (distutils -> setup tools maybe? or wheels -> eggs?)
9.4.1. Poetry
Apparently after setup you just do two things to make a package:
poetry build -f wheel
poetry publish
But before that you have to enter the poetry gestalt, starting at:
https://python-poetry.org/docs/basic-usage/
So I did this on a CentOS7 machine with python3 installed from the rh-python36 software collection.
I did the following:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
poetry --version # returns 1.1.11 at this time, 2021-10-26
mkdir -p ~/repo/
cd ~/repo/
poetry new try-poetry
cd try-poetry
git init .
git add `find . -type f`
git commit -a -m "initial files created by potry new"
Now you can look at pyproject.toml to see the project metadata.
Eventually I run in to the fact that if I do:
poetry add cartopy@0.19.0.post1
poetry add matplotlib
poetry run try_poetry/try_poetry.py
it works, but it required that I do an “apt install” of the projection library.
So I set this aside for now, since it looks like I will really need conda stuff.
9.4.2. miniconda inside an RPM
I’d like to use conda, since it handles the building of C/FORTRAN libraries.
Links:
https://stackoverflow.com/questions/44280661/reallocate-full-conda-installation
https://github.com/pelson/conda-rpms
try things like:
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > Miniconda.sh
bash Miniconda.sh -b -p $HOME/conda
conda create -y --prefix /tmp/try_conda_rpm
# conda init bash
conda activate /tmp/try_conda_rpm
conda install -y cartopy --prefix /tmp/try_conda_rpm
9.5. Digging deep into some of them
9.6. Additional resources
Here are various links that I started with, to then come up with the recipes I tried.