Packaging the pysteps project#

The Python Package Index (PyPI) is a software repository for the Python programming language. PyPI helps you find and install software developed and shared by the Python community.

The following guide to package pysteps was adapted from the PyPI official documentation.

Generating the source distribution#

The first step is to generate a source distribution (sdist) for the pysteps library. These are archives that are uploaded to the Package Index and can be installed by pip.

To create the sdist package we need the setuptools package installed.

Then, from the root folder of the pysteps source run:

python setup.py sdist

Once this command is completed, it should generate a tar.gz (source archive) file the dist directory:

dist/
  pysteps-a.b.c.tar.gz

where a.b.c denote the version number.

Uploading the source distribution to the archive#

The last step is to upload your package to the Python Package Index.

Important

Before we actually upload the distribution to the Python Index, we will test it in Test PyPI. Test PyPI is a separate instance of the package index that allows us to try the distribution without affecting the real index (PyPi). Because TestPyPI has a separate database from the actual PyPI, you’ll need a separate user account for specifically for TestPyPI. You can register your account in https://test.pypi.org/account/register/.

Once you are registered, you can use twine to upload the distribution packages. Alternatively, the package can be uploaded manually from the Test PyPI page.

If Twine is not installed, you can install it by running pip install twine or conda install twine.

Test PyPI#

To upload the recently created source distribution (dist/pysteps-a.b.c.tar.gz) under the dist directory run:

twine upload --repository-url https://test.pypi.org/legacy/ dist/pysteps-a.b.c.tar.gz

where a.b.c denote the version number.

You will be prompted for the username and password you registered with Test PyPI. After the command completes, you should see output similar to this:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading pysteps-a.b.c.tar.gz
100%|█████████████████████| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

Once uploaded your package should be viewable on TestPyPI, for example, https://test.pypi.org/project/pysteps

Test the uploaded package#

Before uploading the package to the official Python Package Index, test that the package can be installed using pip.

Automatic test#

The simplest way to hat the package can be installed using pip is using tox and the tox-conda plugin (conda needed). To install these packages activate your conda development environment and run:

conda install -c conda-forge tox tox-conda

Then, to test the installation in a minimal and an environment with all the dependencies (full env), run:

tox -r -e pypi_test        # Test the installation in a minimal env
tox -r -e pypi_test_full   # Test the installation in an full env

Manual test#

To manually test the installation on new environment, create a copy of the basic development environment using the environment_dev.yml file in the root folder of the pysteps project:

conda env create -f environment_dev.yml -n pysteps_test

Then we activate the environment:

source activate pysteps_test

or:

conda activate pysteps_test

If the environment pysteps_test was already created, remove any version of pysteps already installed:

pip uninstall pysteps

Now, install the pysteps package from test.pypi.org. Since not all the dependecies are available in the Test PyPI repository, we need to add the official repo as an extra index to pip. By doing so, pip will look first in the Test PyPI index and then in the official PyPI:

pip install --no-cache-dir --index-url https://test.pypi.org/simple/  --extra-index-url=https://pypi.org/simple/ pysteps

To test that the installation was successful, from a folder different than the pysteps source, run:

pytest --pyargs pysteps

If any test didn’t pass, check the sources or consider creating a new release fixing those bugs.

Upload package to PyPi#

Once the sdist package was tested, we can safely upload it to the Official PyPi repository with:

twine upload dist/pysteps-a.b.c.tar.gz

Now, pysteps can be installed by simply running:

pip install pysteps

As an extra sanity measure, it is recommended to test the pysteps package installed from the Official PyPi repository (instead of the test PyPi).

Automatic test#

Similarly to the Test the uploaded package section, to test the installation from PyPI in a clean environment, run:

tox -r -e pypi

Manual test#

Follow test instructions in Test PyPI section.