Packaging the pysteps project
Packaging the pysteps project#
The Python Package Index (PyPI) is a software repository for the Python programming language. PyPI helps you find and install software developed and shared by the Python community.
The following guide to package pysteps was adapted from the PyPI official documentation.
Generating the source distribution#
The first step is to generate a source distribution (sdist) for the pysteps library. These are archives that are uploaded to the Package Index and can be installed by pip.
To create the sdist package we need the setuptools package installed.
Then, from the root folder of the pysteps source run:
python setup.py sdist
Once this command is completed, it should generate a tar.gz (source archive) file the dist directory:
where a.b.c denote the version number.
Uploading the source distribution to the archive#
The last step is to upload your package to the Python Package Index.
Before we actually upload the distribution to the Python Index, we will test it in Test PyPI. Test PyPI is a separate instance of the package index that allows us to try the distribution without affecting the real index (PyPi). Because TestPyPI has a separate database from the actual PyPI, you’ll need a separate user account for specifically for TestPyPI. You can register your account in https://test.pypi.org/account/register/.
Once you are registered, you can use twine to upload the distribution packages. Alternatively, the package can be uploaded manually from the Test PyPI page.
If Twine is not installed, you can install it by running
pip install twine or
conda install twine.
To upload the recently created source distribution (dist/pysteps-a.b.c.tar.gz) under the dist directory run:
twine upload --repository-url https://test.pypi.org/legacy/ dist/pysteps-a.b.c.tar.gz
where a.b.c denote the version number.
You will be prompted for the username and password you registered with Test PyPI. After the command completes, you should see output similar to this:
Uploading distributions to https://test.pypi.org/legacy/ Enter your username: [your username] Enter your password: Uploading pysteps-a.b.c.tar.gz 100%|█████████████████████| 4.25k/4.25k [00:01<00:00, 3.05kB/s]
Once uploaded your package should be viewable on TestPyPI, for example, https://test.pypi.org/project/pysteps
Test the uploaded package#
Before uploading the package to the official Python Package Index, test that the package can be installed using pip.
The simplest way to hat the package can be installed using pip is using tox and the tox-conda plugin (conda needed). To install these packages activate your conda development environment and run:
conda install -c conda-forge tox tox-conda
Then, to test the installation in a minimal and an environment with all the dependencies (full env), run:
tox -r -e pypi_test # Test the installation in a minimal env tox -r -e pypi_test_full # Test the installation in an full env
To manually test the installation on new environment, create a copy of the basic development environment using the environment_dev.yml file in the root folder of the pysteps project:
conda env create -f environment_dev.yml -n pysteps_test
Then we activate the environment:
source activate pysteps_test
conda activate pysteps_test
If the environment pysteps_test was already created, remove any version of pysteps already installed:
pip uninstall pysteps
Now, install the pysteps package from test.pypi.org. Since not all the dependecies are available in the Test PyPI repository, we need to add the official repo as an extra index to pip. By doing so, pip will look first in the Test PyPI index and then in the official PyPI:
pip install --no-cache-dir --index-url https://test.pypi.org/simple/ --extra-index-url=https://pypi.org/simple/ pysteps
To test that the installation was successful, from a folder different than the pysteps source, run:
pytest --pyargs pysteps
If any test didn’t pass, check the sources or consider creating a new release fixing those bugs.
Upload package to PyPi#
Once the sdist package was tested, we can safely upload it to the Official PyPi repository with:
twine upload dist/pysteps-a.b.c.tar.gz
Now, pysteps can be installed by simply running:
pip install pysteps
As an extra sanity measure, it is recommended to test the pysteps package installed from the Official PyPi repository (instead of the test PyPi).
Similarly to the Test the uploaded package section, to test the installation from PyPI in a clean environment, run:
tox -r -e pypi
Follow test instructions in Test PyPI section.