pySTEPS – The nowcasting initiative

Pysteps is a community-driven initiative for developing and maintaining an easy to use, modular, free and open source Python framework for short-term ensemble prediction systems.

The focus is on probabilistic nowcasting of radar precipitation fields, but pysteps is designed to allow a wider range of uses.

Pysteps is actively developed on GitHub, while a more thorough description of pysteps is available in the pysteps reference publication:

Pulkkinen, S., D. Nerini, A. Perez Hortal, C. Velasco-Forero, U. Germann, A. Seed, and L. Foresti, 2019: Pysteps: an open-source Python library for probabilistic precipitation nowcasting (v1.0). Geosci. Model Dev. Discuss., doi:10.5194/gmd-2019-94, in review. [source]

Documentation

The documentation is separated in three big branches, intended for different audiences.

User guide

This section is intended for new pysteps users. It provides an introductory overview to the pysteps package, explains how to install it and make use of the most important features.

pySTEPS reference

Comprehensive description of all the modules and functions available in pysteps.

pySTEPS developer guide

Resources and guidelines for pysteps developers and contributors.

Contents

User guide

This guide is gives an introductory overview to the pySTEPS package. It explains how to install and make use of the most important features.

For detailed reference documentation of the modules and functions available in the package see the pySTEPS reference.

** Under development **

Installing pysteps

Dependencies

The pysteps package needs the following dependencies

Additionally, the following packages can be installed for better computational efficiency:

  • dask and toolz (for code parallelization)
  • pyfftw (for faster FFT computation)

Other optional dependencies include:

Install from source

The recommended way to install pysteps from source is using pip to adhere to the [PEP517 standards](https://www.python.org/dev/peps/pep-0517/). Using pip instead of setup.py guarantees that all the package dependencies are properly handled during the installation process.

OSX users

pySTEPS uses Cython extensions that need to be compiled with multi-threading support enabled. The default Apple Clang compiler does not support OpenMP, so using the default compiler would have disabled multi-threading and you will get the following error during the installation:

clang: error: unsupported option '-fopenmp'
error: command 'gcc' failed with exit status 1

To solve this issue, obtain the lastest gcc version with Homebrew that has multi-threading enabled:

brew install gcc

To make sure that the installer uses the homebrew’s gcc, export the following environmental variables in the terminal (supposing that gcc version 8 was installed):

export CC=gcc-8
export CXX=g++-8

First, check that the homebrew’s gcc is detected:

which gcc-8

This should point to the homebrew’s gcc installation.

Under certain circumstances, Homebrew does not add the symbolic links for the gcc executables under /usr/local/bin. If that is the case, specify the CC and CCX variables using the full path to the homebrew installation. For example:

export CC=/usr/local/Cellar/gcc/8.3.0/bin/gcc-8
export CXX=/usr/local/Cellar/gcc/8.3.0/bin/g++-8

Then, you can continue with the normal installation procedure described next.

Installation

The latest pysteps version in the repository can be installed using pip by simply running in a terminal:

pip install git+https://github.com/pySTEPS/pysteps

Or, from a local copy of the repo (global installation):

git clone https://github.com/pySTEPS/pysteps
cd pysteps
pip install .

The above commands will install the latest version of the master branch, which is constantly under development.

Development mode

The latest version can also be installed in Development Mode, i.e., in such a way that the project appears to be installed, but yet is still editable from the source tree:

pip install -e <path to local pysteps repo>
Setting up the user-defined configuration file

The pysteps package allows the users to customize the default settings and configuration. The configuration parameters used by default are loaded from a user-defined JSON file and then stored in the pysteps.rcparams AttrDict.

The pySTEPS configuration file (pystepsrc)

The pysteps package allows the users to customize the default settings and configuration. The configuration parameters used by default are loaded from a user-defined JSON file and then stored in the pysteps.rcparams AttrDict.

The configuration parameters can be accessed as attributes or as items in a dictionary. For e.g., to retrieve the default parameters the following ways are equivalent:

import pysteps

# Retrieve the colorscale for plots
colorscale = pysteps.rcparams['plot']['colorscale']
colorscale = pysteps.rcparams.plot.colorscale

# Retrieve the the root directory of the fmi data
pysteps.rcparams['data_sources']['fmi']['root_path']
pysteps.rcparams.data_sources.fmi.root_path

A less wordy alternative:

from pysteps import rcparams
colorscale = rcparams['plot']['colorscale']
colorscale = rcparams.plot.colorscale

fmi_root_path = rcparams['data_sources']['fmi']['root_path']
fmi_root_path = rcparams.data_sources.fmi.root_path

When the pysteps package imported, it looks for pystepsrc file in the following order:

  • $PWD/pystepsrc : Looks for the file in the current directory
  • $PYSTEPSRC : If the system variable $PYSTEPSRC is defined and it points to a file, it is used.
  • $PYSTEPSRC/pystepsrc : If $PYSTEPSRC points to a directory, it looks for the pystepsrc file inside that directory.
  • $HOME/.pysteps/pystepsrc (unix and Mac OS X) : If the system variable $HOME is defined, it looks for the configuration file in this path.
  • $USERPROFILE/pysteps/pystepsrc (windows only): It looks for the configuration file in the pysteps directory located user’s home directory.
  • Lastly, it looks inside the library in pysteps/pystepsrc for a system-defined copy.

The recommended method to set-up the configuration files is to edit a copy of the default pystepsrc file that is distributed with the package and place that copy inside the user home folder. See the instructions below.

Setting up the user-defined configuration file
Linux and OSX users

For Linux and OSX users, the recommended way to customize the pysteps configuration is place the pystepsrc parameters file in the users home folder ${HOME} in the following path: ${HOME}/.pysteps/pystepsrc

To steps to setup up the configuration file in the home directory first we need to create the directory if it does not exist. In a terminal, run:

$ mkdir -p ${HOME}/.pysteps

The next step is to find the location of the library’s pystepsrc file being actually used. When we import pysteps in a python interpreter, the configuration file loaded is shown:

import pysteps
"Pysteps configuration file found at: /path/to/pysteps/library/pystepsrc"

Then we copy the library’s default rc file to that directory:

$ cp /path/to/pysteps/library/pystepsrc ${HOME}/.pysteps/pystepsrc

Edit the file with the text editor of your preference and change the default configurations with your preferences.

Finally, check that the new updated file is being loaded by the library:

import pysteps
"Pysteps configuration file found at: /home/user_name/.pysteps/pystepsrc"
Windows

For windows users, the recommended way to customize the pySTEPS configuration is place the pystepsrc parameters file in the users folder (defined in the %USERPROFILE% environment variable) in the following path: %USERPROFILE%/pysteps/pystepsrc

To steps to setup up the configuration file in the home directory first we need to create the directory if it does not exist. In a terminal, run:

$ mkdir -p %USERPROFILE%\pysteps

The next step is to find the location of the library’s pystepsrc file being actually used. When we import pysteps in a python interpreter, the configuration file loaded is shown:

import pysteps
"Pysteps configuration file found at: C:\path\to\pysteps\library\pystepsrc"

Then we copy the library’s default rc file to that directory:

$ cp C:\path\to\pysteps\library\pystepsrc %USERPROFILE%\pysteps\pystepsrc

Edit the file with the text editor of your preference and change the default configurations with your preferences.

Finally, check that the new updated file is being loaded by the library:

import pysteps
"Pysteps configuration file found at: C:\User\Profile\.pysteps\pystepsrc"
More
Example of pystepsrc file

Below you can find the default pystepsrc file. The lines starting with “//” are comments and they are ignored.

// pysteps configuration
{
    // "silent_import" : whether to suppress the initial pysteps message
    "silent_import": false,
    "outputs": {
        // path_outputs : path where to save results (figures, forecasts, etc)
        "path_outputs": "./"
    },
    "plot": {
        // "motion_plot" : "streamplot" or "quiver"
        "motion_plot": "quiver",
        // "colorscale" :  "BOM-RF3", "pysteps" or "STEPS-BE"
        "colorscale": "pysteps"
    },
    "data_sources": {
        "bom": {
            "root_path": "./radar/bom",
            "path_fmt": "prcp-cscn/2/%Y/%m/%d",
            "fn_pattern": "2_%Y%m%d_%H%M00.prcp-cscn",
            "fn_ext": "nc",
            "importer": "bom_rf3",
            "timestep": 6,
            "importer_kwargs": {
                "gzipped": true
            }
        },
        "fmi": {
            "root_path": "./radar/fmi",
            "path_fmt": "%Y%m%d",
            "fn_pattern": "%Y%m%d%H%M_fmi.radar.composite.lowest_FIN_SUOMI1",
            "fn_ext": "pgm.gz",
            "importer": "fmi_pgm",
            "timestep": 5,
            "importer_kwargs": {
                "gzipped": true
            }
        },
        "mch": {
            "root_path": "./radar/mch",
            "path_fmt": "%Y%m%d",
            "fn_pattern": "AQC%y%j%H%M?_00005.801",
            "fn_ext": "gif",
            "importer": "mch_gif",
            "timestep": 5,
            "importer_kwargs": {
                "product": "AQC",
                "unit": "mm",
                "accutime": 5
            }
        },
        "opera": {
            "root_path": "./radar/OPERA",
            "path_fmt": "%Y%m%d",
            "fn_pattern": "T_PAAH21_C_EUOC_%Y%m%d%H%M%S",
            "fn_ext": "hdf",
            "importer": "opera_hdf5",
            "timestep": 15,
            "importer_kwargs": {}
        },
        "knmi": {
            "root_path": "./radar/KNMI",
            "path_fmt": "%Y/%m",
            "fn_pattern": "RAD_NL25_RAP_5min_%Y%m%d%H%M",
            "fn_ext": "h5",
            "importer": "knmi_hdf5",
            "timestep": 5,
            "importer_kwargs": {
                "accutime": 5,
                "qty": "ACRR",
                "pixelsize": 1000.0
            }
        }
    }
}

Installing the Example data

The examples scripts in the user guide as well as the pySTEPS build-in tests use the example radar data available in a separate repository: pysteps-data.

The data must be downloaded manually into your computer and the ref:pystepsrc” file need to configured to point to that example data.

First, download the data from the repository by clicking here.

Unzip the data into a folder of your preference. Once the data is unzip, the directory structure looks like this:

pysteps-data
|
├── radar
      ├── KNMI
      ├── OPERA
      ├── bom
      ├── fmi
      ├── mch

Now we need to update the pystepsrc file for the each of the data_sources to point to these directories, as described in The pySTEPS configuration file (pystepsrc).

pySTEPS reference

Release:1.1.0
Date:Oct 24, 2019

This page gives an comprehensive description of all the modules and functions available in pySTEPS.

pysteps.cascade

Methods for constructing bandpass filters and decomposing 2d precipitation fields into different spatial scales.

pysteps.cascade.interface

Interface for the cascade module.

get_method(name) Return a callable function for the bandpass filter or decomposition method corresponding to the given name.
pysteps.cascade.bandpass_filters

Bandpass filters for separating different spatial scales from two-dimensional images in the frequency domain.

The methods in this module implement the following interface:

filter_xxx(shape, n, optional arguments)

where shape is the shape of the input field, respectively, and n is the number of frequency bands to use.

The output of each filter function is a dictionary containing the following key-value pairs:

Key Value
weights_1d 2d array of shape (n, r) containing 1d filter weights for each frequency band k=1,2,…,n
weights_2d 3d array of shape (n, M, int(N/2)+1) containing the 2d filter weights for each frequency band k=1,2,…,n
central_freqs 1d array of shape n containing the central frequencies of the filters

where r = int(max(N, M)/2)+1

By default, the filter weights are normalized so that for any Fourier wavenumber they sum to one.

Available filters
filter_uniform(shape, n) A dummy filter with one frequency band covering the whole domain.
filter_gaussian(shape, n[, l_0, …]) Implements a set of Gaussian bandpass filters in logarithmic frequency scale.
pysteps.cascade.decomposition

Methods for decomposing two-dimensional images into multiple spatial scales.

The methods in this module implement the following interface:

decomposition_xxx(X, filter, **kwargs)

where X is the input field and filter is a dictionary returned by a filter method implemented in pysteps.cascade.bandpass_filters. Optional parameters can be passed in the keyword arguments. The output of each method is a dictionary with the following key-value pairs:

Key Value
cascade_levels three-dimensional array of shape (k,m,n), where k is the number of cascade levels and the input fields have shape (m,n)
means list of mean values for each cascade level
stds list of standard deviations for each cascade level
Available methods
decomposition_fft(X, filter, \*\*kwargs) Decompose a 2d input field into multiple spatial scales by using the Fast Fourier Transform (FFT) and a bandpass filter.

pysteps.extrapolation

Extrapolation module functions and interfaces.

pysteps.extrapolation.interface

The functions in the extrapolation module implement the following interface:

extrapolate(extrap, precip, velocity, num_timesteps,
            outval=np.nan, **keywords)

where extrap is an extrapolator object returned by the initialize function, precip is a (m,n) array with input precipitation field to be advected and velocity is a (2,m,n) array containing the x- and y-components of the m x n advection field. num_timesteps is an integer specifying the number of time steps to extrapolate. The optional argument outval specifies the value for pixels advected from outside the domain. Optional keyword arguments that are specific to a given extrapolation method are passed as a dictionary.

The output of each method is an array R_e that includes the time series of extrapolated fields of shape (num_timesteps, m, n).

get_method(name) Return two-element tuple for the extrapolation method corresponding to the given name.
eulerian_persistence(precip, velocity, …) A dummy extrapolation method to apply Eulerian persistence to a two-dimensional precipitation field.
pysteps.extrapolation.semilagrangian

Implementation of the semi-Lagrangian method of Germann et al (2002). [GZ2002]

extrapolate(precip, velocity, num_timesteps) Apply semi-Lagrangian backward extrapolation to a two-dimensional precipitation field.

pysteps.io

Methods for browsing data archives, reading 2d precipitation fields and writing forecasts into files.

pysteps.io.interface

Interface for the io module.

get_method(name, method_type) Return a callable function for the method corresponding to the given name.
pysteps.io.archive

Utilities for finding archived files that match the given criteria.

find_by_date(date, root_path, path_fmt, …) List input files whose timestamp matches the given date.
pysteps.io.importers

Methods for importing files containing two-dimensional radar mosaics.

The methods in this module implement the following interface:

import_xxx(filename, optional arguments)

where xxx is the name (or abbreviation) of the file format and filename is the name of the input file.

The output of each method is a three-element tuple containing a two-dimensional radar mosaic, the corresponding quality field and a metadata dictionary. If the file contains no quality information, the quality field is set to None. Pixels containing missing data are set to nan.

The metadata dictionary contains the following recommended key-value pairs:

Key Value
projection PROJ.4-compatible projection definition
x1 x-coordinate of the lower-left corner of the data raster (meters)
y1 y-coordinate of the lower-left corner of the data raster (meters)
x2 x-coordinate of the upper-right corner of the data raster (meters)
y2 y-coordinate of the upper-right corner of the data raster (meters)
xpixelsize grid resolution in x-direction (meters)
ypixelsize grid resolution in y-direction (meters)
yorigin a string specifying the location of the first element in the data raster w.r.t. y-axis: ‘upper’ = upper border ‘lower’ = lower border
institution name of the institution who provides the data
unit the physical unit of the data: ‘mm/h’, ‘mm’ or ‘dBZ’
transform the transformation of the data: None, ‘dB’, ‘Box-Cox’ or others
accutime the accumulation time in minutes of the data, float
threshold the rain/no rain threshold with the same unit, transformation and accutime of the data.
zerovalue the value assigned to the no rain pixels with the same unit, transformation and accutime of the data.
zr_a the Z-R constant a in Z = a*R**b
zr_b the Z-R exponent b in Z = a*R**b
Available Importers
import_bom_rf3(filename, \*\*kwargs) Import a NetCDF radar rainfall product from the BoM Rainfields3.
import_fmi_geotiff(filename, \*\*kwargs) Import a reflectivity field (dBZ) from an FMI GeoTIFF file.
import_fmi_pgm(filename, \*\*kwargs) Import a 8-bit PGM radar reflectivity composite from the FMI archive.
import_mch_gif(filename, product, unit, accutime) Import a 8-bit gif radar reflectivity composite from the MeteoSwiss archive.
import_mch_hdf5(filename, \*\*kwargs) Import a precipitation field (and optionally the quality field) from a MeteoSwiss HDF5 file conforming to the ODIM specification.
import_mch_metranet(filename, product, unit, …) Import a 8-bit bin radar reflectivity composite from the MeteoSwiss archive.
import_opera_hdf5(filename, \*\*kwargs) Import a precipitation field (and optionally the quality field) from an OPERA HDF5 file conforming to the ODIM specification.
import_knmi_hdf5(filename, \*\*kwargs) Import a precipitation or reflectivity field (and optionally the quality field) from a HDF5 file conforming to the KNMI Data Centre specification.
pysteps.io.nowcast_importers

Methods for importing nowcast files.

The methods in this module implement the following interface:

import_xxx(filename, optional arguments)

where xxx is the name (or abbreviation) of the file format and filename is the name of the input file.

The output of each method is a two-element tuple containing the nowcast array and a metadata dictionary.

The metadata dictionary contains the following mandatory key-value pairs:

Key Value
projection PROJ.4-compatible projection definition
x1 x-coordinate of the lower-left corner of the data raster (meters)
y1 y-coordinate of the lower-left corner of the data raster (meters)
x2 x-coordinate of the upper-right corner of the data raster (meters)
y2 y-coordinate of the upper-right corner of the data raster (meters)
xpixelsize grid resolution in x-direction (meters)
ypixelsize grid resolution in y-direction (meters)
yorigin a string specifying the location of the first element in the data raster w.r.t. y-axis: ‘upper’ = upper border ‘lower’ = lower border
institution name of the institution who provides the data
timestep time step of the input data (minutes)
unit the physical unit of the data: ‘mm/h’, ‘mm’ or ‘dBZ’
transform the transformation of the data: None, ‘dB’, ‘Box-Cox’ or others
accutime the accumulation time in minutes of the data, float
threshold the rain/no rain threshold with the same unit, transformation and accutime of the data.
zerovalue it is the value assigned to the no rain pixels with the same unit, transformation and accutime of the data.
Available Nowcast Importers
import_netcdf_pysteps(filename, \*\*kwargs) Read a nowcast or a nowcast ensemble from a NetCDF file conforming to the CF 1.7 specification.
pysteps.io.exporter

Methods for exporting forecasts of 2d precipitation fields into various file formats.

Each exporter method in this module has its own initialization function that implements the following interface:

initialize_forecast_exporter_xxx(filename, startdate, timestep,
                                 num_timesteps, shape, num_ens_members,
                                 metadata, incremental=None)

where xxx is the name (or abbreviation) of the file format.

This function creates the file and writes the metadata. The datasets are written by calling pysteps.io.exporters.export_forecast_dataset(), and the file is closed by calling pysteps.io.exporters.close_forecast_file().

The arguments in the above are defined as follows:

Argument Type/values Description
filename str name of the output file
startdate datetime.datetime start date of the forecast
timestep int time step of the forecast (minutes)
n_timesteps int number of time steps in the forecast this argument is ignored if incremental is set to ‘timestep’.
shape tuple two-element tuple defining the shape (height,width) of the forecast grids
n_ens_members int number of ensemble members in the forecast. This argument is ignored if incremental is set to ‘member’
metadata dict metadata dictionary containing the projection,x1,x2,y1,y2 and unit attributes described in the documentation of pysteps.io.importers
incremental {None, ‘timestep’, ‘member’} Allow incremental writing of datasets into the netCDF file the available options are: ‘timestep’ = write a forecast or a forecast ensemble for a given time step ‘member’ = write a forecast sequence for a given ensemble member

The return value is a dictionary containing an exporter object. This can be used with pysteps.io.exporters.export_forecast_dataset() to write datasets into the given file format.

Available Exporters
initialize_forecast_exporter_kineros(…[, …]) Initialize a KINEROS2 Rainfall .pre file as specified in https://www.tucson.ars.ag.gov/kineros/.
initialize_forecast_exporter_netcdf(…[, …]) Initialize a netCDF forecast exporter.
Generic functions
export_forecast_dataset(F, exporter) Write a forecast array into a file.
close_forecast_file(exporter) Close the file associated with a forecast exporter.
pysteps.io.readers

Module with the reader functions.

read_timeseries(inputfns, importer, \*\*kwargs) Read a time series of input files using the methods implemented in the pysteps.io.importers module and stack them into a 3d array of shape (num_timesteps, height, width).

pysteps.motion

Implementations of optical flow methods.

pysteps.motion.interface

Interface for the motion module. It returns a callable optical flow routine for computing the motion field.

The methods in the motion module implement the following interface:

motion_method(precip, **keywords)

where precip is a (T,m,n) array containing a sequence of T two-dimensional input images of shape (m,n). The first dimension represents the images time dimension and the value of T depends on the type of the method.

The output is a three-dimensional array (2,m,n) containing the dense x- and y-components of the motion field in units of pixels / timestep as given by the input array R.

get_method(name) Return a callable function for the optical flow method corresponding to the given name.
pysteps.motion.constant

Implementation of a constant advection field estimation by maximizing the correlation between two images.

constant(R, \*\*kwargs) Compute a constant advection field by finding a translation vector that maximizes the correlation between two successive images.
pysteps.motion.darts

Implementation of the DARTS algorithm.

DARTS(input_images, \*\*kwargs) Compute the advection field from a sequence of input images by using the DARTS method.
pysteps.motion.lucaskanade

The Lucas-Kanade (LK) local feature tracking module.

This module implements the interface to the local Lucas-Kanade routine available in OpenCV.

For its dense method, it additionally interpolates the sparse vectors over a regular grid to return a motion field.

dense_lucaskanade(input_images[, lk_kwargs, …]) Run the Lucas-Kanade optical flow routine and interpolate the motion vectors.
track_features(prvs_image, next_image, points) Interface to the OpenCV Lucas-Kanade features tracking algorithm (cv.calcOpticalFlowPyrLK).
pysteps.motion.proesmans

Implementation of the anisotropic diffusion method of Proesmans et al. (1994).

proesmans(input_images[, lam, num_iter, …]) Implementation of the anisotropic diffusion method of Proesmans et al.
pysteps.motion.vet

Variational Echo Tracking (VET) Module

This module implements the VET algorithm presented by Laroche and Zawadzki (1995) and used in the McGill Algorithm for Prediction by Lagrangian Extrapolation (MAPLE) described in Germann and Zawadzki (2002).

The morphing and the cost functions are implemented in Cython and parallelized for performance.

vet(input_images[, sectors, smooth_gain, …]) Variational Echo Tracking Algorithm presented in Laroche and Zawadzki (1995) and used in the McGill Algorithm for Prediction by Lagrangian Extrapolation (MAPLE) described in Germann and Zawadzki (2002).
vet_cost_function(sector_displacement_1d, …)
vet_cost_function_gradient(\*args, \*\*kwargs) Compute the vet cost function gradient.
morph(image, displacement[, gradient]) Morph image by applying a displacement field (Warping).
round_int(scalar) Round number to nearest integer.
ceil_int(scalar) Round number to nearest integer.
get_padding(dimension_size, sectors) Get the padding at each side of the one dimensions of the image so the new image dimensions are divided evenly in the number of sectors specified.

pysteps.noise

Implementation of deterministic and ensemble nowcasting methods.

pysteps.noise.interface

Interface for the noise module.

get_method(name) Return two callable functions to initialize and generate 2d perturbations of precipitation or velocity fields.
pysteps.noise.fftgenerators

Methods for noise generators based on FFT filtering of white noise.

The methods in this module implement the following interface for filter initialization depending on their parametric or nonparametric nature:

initialize_param_2d_xxx_filter(X, **kwargs)

or:

initialize_nonparam_2d_xxx_filter(X, **kwargs)

where X is an array of shape (m, n) or (t, m, n) that defines the target field and optional parameters are supplied as keyword arguments.

The output of each initialization method is a dictionary containing the keys F and input_shape. The first is a two-dimensional array of shape (m, int(n/2)+1) that defines the filter. The second one is the shape of the input field for the filter.

The methods in this module implement the following interface for the generation of correlated noise:

generate_noise_2d_xxx_filter(F, randstate=np.random, seed=None, **kwargs)

where F (m, n) is a filter returned from the corresponding initialization method, and randstate and seed can be used to set the random generator and its seed. Additional keyword arguments can be included as a dictionary.

The output of each generator method is a two-dimensional array containing the field of correlated noise cN of shape (m, n).

initialize_param_2d_fft_filter(X, \*\*kwargs) Takes one ore more 2d input fields, fits two spectral slopes, beta1 and beta2, to produce one parametric, global and isotropic fourier filter.
initialize_nonparam_2d_fft_filter(X, \*\*kwargs) Takes one ore more 2d input fields and produces one non-paramtric, global and anasotropic fourier filter.
initialize_nonparam_2d_nested_filter(X[, …]) Function to compute the local Fourier filters using a nested approach.
initialize_nonparam_2d_ssft_filter(X, \*\*kwargs) Function to compute the local Fourier filters using the Short-Space Fourier filtering approach.
generate_noise_2d_fft_filter(F[, randstate, …]) Produces a field of correlated noise using global Fourier filtering.
generate_noise_2d_ssft_filter(F[, …]) Function to compute the locally correlated noise using a nested approach.
pysteps.noise.motion

Methods for generating perturbations of two-dimensional motion fields.

The methods in this module implement the following interface for initialization:

inizialize_xxx(V, pixelsperkm, timestep, optional arguments)

where V (2,m,n) is the motion field and pixelsperkm and timestep describe the spatial and temporal resolution of the motion vectors. The output of each initialization method is a dictionary containing the perturbator that can be supplied to generate_xxx.

The methods in this module implement the following interface for the generation of a motion perturbation field:

generate_xxx(perturbator, t, randstate=np.random, seed=None)

where perturbator is a dictionary returned by an initialize_xxx method. Optional random generator can be specified with the randstate and seed arguments, respectively. The output of each generator method is an array of shape (2,m,n) containing the x- and y-components of the motion vector perturbations, where m and n are determined from the perturbator.

get_default_params_bps_par() Return a tuple containing the default velocity perturbation parameters given in [BPS2006] for the parallel component.
get_default_params_bps_perp() Return a tuple containing the default velocity perturbation parameters given in [BPS2006] for the perpendicular component.
initialize_bps(V, pixelsperkm, timestep[, …]) Initialize the motion field perturbator described in [BPS2006].
generate_bps(perturbator, t) Generate a motion perturbation field by using the method described in [BPS2006].
pysteps.noise.utils

Miscellaneous utility functions related to generation of stochastic perturbations.

compute_noise_stddev_adjs(R, R_thr_1, …[, …]) Apply a scale-dependent adjustment factor to the noise fields used in STEPS.

pysteps.nowcasts

Implementation of deterministic and ensemble nowcasting methods.

pysteps.nowcasts.interface

Interface for the nowcasts module. It returns a callable function for computing nowcasts.

The methods in the nowcasts module implement the following interface:

forecast(precip, velocity, num_timesteps, **keywords)

where precip is a (m,n) array with input precipitation field to be advected and velocity is a (2,m,n) array containing the x- and y-components of the m x n advection field. num_timesteps is an integer specifying the number of time steps to forecast. The interface accepts optional keyword arguments specific to the given method.

The output depends on the type of the method. For deterministic methods, the output is a three-dimensional array of shape (num_timesteps,m,n) containing a time series of nowcast precipitation fields. For stochastic methods that produce an ensemble, the output is a four-dimensional array of shape (num_ensemble_members,num_timesteps,m,n). The time step of the output is taken from the inputs.

get_method(name) Return a callable function for computing nowcasts.
pysteps.nowcasts.extrapolation

Implementation of extrapolation-based nowcasting methods.

forecast(precip, velocity, num_timesteps[, …]) Generate a nowcast by applying a simple advection-based extrapolation to the given precipitation field.
pysteps.nowcasts.sprog

Implementation of the S-PROG method described in [Seed2003]

forecast(R, V, n_timesteps[, …]) Generate a nowcast by using the Spectral Prognosis (S-PROG) method.
pysteps.nowcasts.sseps

Implementation of the Short-space ensemble prediction system (SSEPS) method. Essentially, SSEPS is a localized version of STEPS.

For localization we intend the use of a subset of the observations in order to estimate model parameters that are distributed in space. The short-space approach used in [NBSG2017] is generalized to the whole nowcasting system. This essenially boils down to a moving window localization of the nowcasting procedure, whereby all parameters are estimated over a subdomain of prescribed size.

forecast(R, metadata, V, n_timesteps[, …]) Generate a nowcast ensemble by using the Short-space ensemble prediction system (SSEPS) method.
pysteps.nowcasts.steps

Implementation of the STEPS stochastic nowcasting method as described in [Seed2003], [BPS2006] and [SPN2013].

forecast(R, V, n_timesteps[, n_ens_members, …]) Generate a nowcast ensemble by using the Short-Term Ensemble Prediction System (STEPS) method.
pysteps.nowcasts.utils

Module with common utilities used by nowcasts methods.

print_ar_params(PHI) Print the parameters of an AR(p) model.
print_corrcoefs(GAMMA) Print the parameters of an AR(p) model.
stack_cascades(R_d, n_levels[, donorm]) Stack the given cascades into a larger array.
recompose_cascade(R, mu, sigma) Recompose a cascade by inverting the normalization and summing the cascade levels.

pysteps.postprocessing

Methods for post-processing of forecasts.

pysteps.postprocessing.ensemblestats

Methods for the computation of ensemble statistics.

mean(X[, ignore_nan, X_thr]) Compute the mean value from a forecast ensemble field.
excprob(X, X_thr[, ignore_nan]) For a given forecast ensemble field, compute exceedance probabilities for the given intensity thresholds.
pysteps.postprocessing.probmatching

Methods for matching the probability distribution of two data sets.

compute_empirical_cdf(bin_edges, hist) Compute an empirical cumulative distribution function from the given histogram.
nonparam_match_empirical_cdf(R, R_trg) Matches the empirical CDF of the initial array with the empirical CDF of a target array.
pmm_init(bin_edges_1, cdf_1, bin_edges_2, cdf_2) Initialize a probability matching method (PMM) object from binned cumulative distribution functions (CDF).
pmm_compute(pmm, x) For a given PMM object and x-coordinate, compute the probability matched value (i.e.
shift_scale(R, f, rain_fraction_trg, …) Find shift and scale that is needed to return the required second_moment and rain area.

pysteps.timeseries

Methods and models for time series analysis.

pysteps.timeseries.autoregression

Methods related to autoregressive AR(p) models.

adjust_lag2_corrcoef1(gamma_1, gamma_2) A simple adjustment of lag-2 temporal autocorrelation coefficient to ensure that the resulting AR(2) process is stationary when the parameters are estimated from the Yule-Walker equations.
adjust_lag2_corrcoef2(gamma_1, gamma_2) A more advanced adjustment of lag-2 temporal autocorrelation coefficient to ensure that the resulting AR(2) process is stationary when the parameters are estimated from the Yule-Walker equations.
ar_acf(gamma[, n]) Compute theoretical autocorrelation function (ACF) from the AR(p) model with lag-l, l=1,2,…,p temporal autocorrelation coefficients.
estimate_ar_params_yw(gamma) Estimate the parameters of an AR(p) model from the Yule-Walker equations using the given set of autocorrelation coefficients.
iterate_ar_model(X, phi[, EPS]) Apply an AR(p) model to a time-series of two-dimensional fields.
pysteps.timeseries.correlation

Methods for computing spatial and temporal correlation of time series of two-dimensional fields.

temporal_autocorrelation(X[, MASK]) Compute lag-l autocorrelation coefficients gamma_l, l=1,2,…,n-1, for a time series of n two-dimensional input fields.

pysteps.utils

Implementation of miscellaneous utility functions.

pysteps.utils.interface

Interface for the utils module.

get_method(name, \*\*kwargs) Return a callable function for the utility method corresponding to the given name.
pysteps.utils.arrays

Utility methods for creating and processing arrays.

compute_centred_coord_array(M, N) Compute a 2D coordinate array, where the origin is at the center.
pysteps.utils.cleansing

Data cleansing routines for pysteps.

decluster(coord, input_array, scale[, …]) Decluster a set of sparse data points by aggregating, that is, taking the median value of all values lying within a certain distance (i.e., a cluster).
detect_outliers(input_array, thr[, coord, …]) Detect outliers in a (multivariate and georeferenced) dataset.
pysteps.utils.conversion

Methods for converting physical units.

to_rainrate(R, metadata[, zr_a, zr_b]) Convert to rain rate [mm/h].
to_raindepth(R, metadata[, zr_a, zr_b]) Convert to rain depth [mm].
to_reflectivity(R, metadata[, zr_a, zr_b]) Convert to reflectivity [dBZ].
pysteps.utils.dimension

Functions to manipulate array dimensions.

aggregate_fields_time(R, metadata, …[, …]) Aggregate fields in time.
aggregate_fields_space(R, metadata, space_window) Upscale fields in space.
aggregate_fields(R, window_size[, axis, method]) Aggregate fields.
clip_domain(R, metadata[, extent]) Clip the field domain by geographical coordinates.
square_domain(R, metadata[, method, inverse]) Either pad or crop a field to obtain a square domain.
pysteps.utils.fft

Interface module for different FFT methods.

get_numpy(shape[, fftn_shape])
get_scipy(shape[, fftn_shape])
get_pyfftw(shape[, fftn_shape, n_threads])
pysteps.utils.images

Image processing routines for pysteps.

ShiTomasi_detection(input_image[, …]) Interface to the OpenCV Shi-Tomasi features detection method to detect corners in an image.
morph_opening(input_image, thr, n) Filter out small scale noise on the image by applying a binary morphological opening, that is, erosion followed by dilation.
pysteps.utils.interpolate

Interpolation routines for pysteps.

rbfinterp2d(coord, input_array, xgrid, ygrid) Fast 2-D grid interpolation of a sparse (multivariate) array using a radial basis function.
pysteps.utils.spectral

Utility methods for processing and analyzing precipitation fields in the Fourier domain.

rapsd(Z[, fft_method, return_freq, d]) Compute radially averaged power spectral density (RAPSD) from the given 2D input field.
remove_rain_norain_discontinuity(R) Function to remove the rain/no-rain discontinuity.
pysteps.utils.transformation

Methods for transforming data values.

boxcox_transform(R[, metadata, Lambda, …]) The one-parameter Box-Cox transformation.
dB_transform(R[, metadata, threshold, …]) Methods to transform precipitation intensities to/from dB units.
NQ_transform(R[, metadata, inverse]) The normal quantile transformation as in Bogner et al (2012).
sqrt_transform(R[, metadata, inverse]) Square-root transform.

pysteps.verification

Methods for verification of deterministic, probabilistic and ensemble forecasts.

pysteps.verification.interface

Interface for the verification module.

get_method(name[, type]) Return a callable function for the method corresponding to the given verification score.
pysteps.verification.detcatscores

Forecast evaluation and skill scores for deterministic categorial (dichotomous) forecasts.

det_cat_fct(pred, obs, thr[, scores, axis]) Calculate simple and skill scores for deterministic categorical (dichotomous) forecasts.
det_cat_fct_init(thr[, axis]) Initialize a contingency table object.
det_cat_fct_accum(contab, pred, obs) Accumulate the frequency of “yes” and “no” forecasts and observations in the contingency table.
det_cat_fct_compute(contab[, scores]) Compute simple and skill scores for deterministic categorical (dichotomous) forecasts from a contingency table object.
pysteps.verification.detcontscores

Forecast evaluation and skill scores for deterministic continuous forecasts.

det_cont_fct(pred, obs[, scores, axis, …]) Calculate simple and skill scores for deterministic continuous forecasts.
det_cont_fct_init([axis, conditioning, thr]) Initialize a verification error object.
det_cont_fct_accum(err, pred, obs) Accumulate the forecast error in the verification error object.
det_cont_fct_compute(err[, scores]) Compute simple and skill scores for deterministic continuous forecasts from a verification error object.
pysteps.verification.ensscores

Evaluation and skill scores for ensemble forecasts.

ensemble_skill(X_f, X_o, metric, \*\*kwargs) Compute mean ensemble skill for a given skill metric.
ensemble_spread(X_f, metric, \*\*kwargs) Compute mean ensemble spread for a given skill metric.
rankhist(X_f, X_o[, X_min, normalize]) Compute a rank histogram counts and optionally normalize the histogram.
rankhist_init(num_ens_members[, X_min]) Initialize a rank histogram object.
rankhist_accum(rankhist, X_f, X_o) Accumulate forecast-observation pairs to the given rank histogram.
rankhist_compute(rankhist[, normalize]) Return the rank histogram counts and optionally normalize the histogram.
pysteps.verification.lifetime

Estimation of precipitation lifetime from a decaying verification score function (e.g. autocorrelation function).

lifetime(X_s, X_t[, rule]) Compute the average lifetime by integrating the correlation function as a function of lead time.
lifetime_init([rule]) Initialize a lifetime object.
lifetime_accum(lifetime, X_s, X_t) Compute the lifetime by integrating the correlation function and accumulate the result into the given lifetime object.
lifetime_compute(lifetime) Compute the average value from the lifetime object.
pysteps.verification.plots

Methods for plotting verification results.

plot_intensityscale(iss[, fig, vmin, vmax, …]) Plot a intensity-scale verification table with a color bar and axis labels.
plot_rankhist(rankhist[, ax]) Plot a rank histogram.
plot_reldiag(reldiag[, ax]) Plot a reliability diagram.
plot_ROC(ROC[, ax, opt_prob_thr]) Plot a ROC curve.
pysteps.verification.probscores

Evaluation and skill scores for probabilistic forecasts.

CRPS(X_f, X_o) Compute the continuous ranked probability score (CRPS).
CRPS_init() Initialize a CRPS object.
CRPS_accum(CRPS, X_f, X_o) Compute the average continuous ranked probability score (CRPS) for a set of forecast ensembles and the corresponding observations and accumulate the result to the given CRPS object.
CRPS_compute(CRPS) Compute the averaged values from the given CRPS object.
reldiag(P_f, X_o, X_min[, n_bins, min_count]) Compute the x- and y- coordinates of the points in the reliability diagram.
reldiag_init(X_min[, n_bins, min_count]) Initialize a reliability diagram object.
reldiag_accum(reldiag, P_f, X_o) Accumulate the given probability-observation pairs into the reliability diagram.
reldiag_compute(reldiag) Compute the x- and y- coordinates of the points in the reliability diagram.
ROC_curve(P_f, X_o, X_min[, n_prob_thrs, …]) Compute the ROC curve and its area from the given ROC object.
ROC_curve_init(X_min[, n_prob_thrs]) Initialize a ROC curve object.
ROC_curve_accum(ROC, P_f, X_o) Accumulate the given probability-observation pairs into the given ROC object.
ROC_curve_compute(ROC[, compute_area]) Compute the ROC curve and its area from the given ROC object.
pysteps.verification.spatialscores

Skill scores for spatial forecasts.

intensity_scale(X_f, X_o, name, thrs[, …]) Compute an intensity-scale verification score.
intensity_scale_init(name, thrs[, scales, …]) Initialize an intensity-scale verification object.
intensity_scale_accum(intscale, X_f, X_o) Compute and update the intensity-scale verification scores.
intensity_scale_compute(intscale) Return the intensity scale matrix.
binary_mse(X_f, X_o, thr[, wavelet]) Compute an intensity-scale verification as the MSE of the binary error.
fss(X_f, X_o, thr, scale) Compute the fractions skill score (FSS) for a deterministic forecast field and the corresponding observation field.
fss_init(thr, scale) Initialize a fractions skill score (FSS) verification object.
fss_accum(fss, X_f, X_o) Accumulate forecast-observation pairs to an FSS object.
fss_compute(fss) Compute the FSS.

pysteps.visualization

Methods for plotting precipitation and motion fields.

pysteps.visualization.animations

Functions to produce animations for pysteps.

animate(R_obs[, nloops, timestamps, R_fct, …]) Function to animate observations and forecasts in pysteps.
pysteps.visualization.motionfields

Functions to plot motion fields.

quiver(UV[, ax, map, geodata, …]) Function to plot a motion field as arrows.
streamplot(UV[, ax, map, geodata, …]) Function to plot a motion field as streamlines.
pysteps.visualization.precipfields

Methods for plotting precipitation fields.

plot_precip_field(R[, type, map, geodata, …]) Function to plot a precipitation intensity or probability field with a colorbar.
get_colormap(type[, units, colorscale]) Function to generate a colormap (cmap) and norm.
pysteps.visualization.spectral

Methods for plotting Fourier spectra.

plot_spectrum1d(fft_freq, fft_power[, …]) Function to plot in log-log a radially averaged Fourier spectrum.
pysteps.visualization.utils

Miscellaneous utility functions for the visualization module.

parse_proj4_string(proj4str) Construct a dictionary from a PROJ.4 projection string.
proj4_to_basemap(proj4str) Convert a PROJ.4 projection string into a dictionary that can be expanded as keyword arguments to mpl_toolkits.basemap.Basemap.__init__.
proj4_to_cartopy(proj4str) Convert a PROJ.4 projection string into a Cartopy coordinate reference system (crs) object.
reproject_geodata(geodata, t_proj4str[, …]) Reproject geodata and optionally create a grid in a new projection.

Indices and tables

Bibliography

pySTEPS developer guide

In this section you can find a series of guidelines and tutorials for contributing to the pySTEPS project.

Contributing to Pysteps

Welcome! pySTEPS is a community-driven initiative for developing and maintaining an easy to use, modular, free and open source Python framework for short-term ensemble prediction systems.

If you haven’t already, take a look at the project’s README.rst file and the pysteps documentation.

There are many ways to contribute to pysteps:

  • contributing bug reports and feature requests
  • contributing documentation
  • code contributions new features or bug fixes
  • contribute with usage examples

Our main forum for discussion is the project’s GitHub issue tracker. This is the right place to start a discussion, report a bug, or request a new feature.

Workflow for code contributions

We welcome all kind of contributions, from documentation updates, a bug fix, or a new feature. If your new feature will take a lot of work, we recommend creating an issue with the enhancement tag to encourage discussions.

We use the usual GitHub pull-request flow, which may be familiar to you if you’ve contributed to other projects on GitHub.

First Time Contributors

If you are interested in helping to improve pysteps, the best way to get started is by looking for “Good First Issue” in the issue tracker.

In a nutshell, the main steps to follow for contributing to pysteps are:

  • Setup the development environment
  • Fork the repository
  • Create a new branch for each contribution
  • Read the Code Style guide
  • Work on your changes
  • Test your changes
  • Push to your fork repository and create a new PR in GitHub.
Setup the Development environment

The recommended way to setup up the developer environment is the Anaconda (commonly referred as Conda). Conda quickly installs, runs and updates packages and their dependencies. It also allows to easily create, save, load and switch between different environments on your local computer.

The developer environment can be created using the environment_dev.yml file in the project’s root directory running the command:

conda env create -f environment_dev.yml

This will create the pysteps_dev environment that can be activated using:

conda activate pysteps_dev

Once the environment is created, the package can be installed in development mode, in such a way that the project appears to be installed, but yet is still editable from the source tree. See instructions in the Installing pysteps section.

Fork the repository

Once you have set the development environment, the next step is creating your local copy of the repository where you will commit your modifications. The steps to follow are:

  1. Set up Git in your computer.

  2. Create a GitHub account (if you don’t have one).

  3. Fork the repository in your GitHub.

  4. Clone local copy of your fork. For example:

    git clone https://github.com/<your-account>/pysteps.git

Done!, now you have a local copy of pysteps git repository. If you are new to GitHub, below you can find a list of useful tutorials:

Create a new branch

As a collaborator, all the new contributions that you want should be done in a new branch under your forked repository. Working on the master branch is reserved for Core Contributors only. Core Contributors are developers that actively work and maintain the repository. They are the only ones who accept pull requests and push commits directly to the pysteps repository.

For more information on how to create and work with branches, see “Branches in a Nutshell” in the Git documentation

Code Style

Although it is not strictly enforced yet, we strongly suggest to follow the PEP8 coding standards. Two popular modules used to check pep8 compliance are pycodestyle and pylint that can be installed using pip:

pip install pylint
pip install pycodestyle

or using anaconda:

conda install pylint
conda install pycodestyle

For further information instructions, the reader is referred to their official documentation.

Coding style summary

For quick reference, these are the most important good coding practices to follow:

  • Always use 4 spaces for indentation (don’t use tabs).

  • Write UTF-8 (add # -*- coding: utf-8 -*- at the top of each file).

  • Max line-length: 79 characters.

  • Always indent wrapped code for readability.

  • Avoid extraneous whitespace.

  • Don’t use whitespace to line up assignment operators (=, :).

  • Spaces around = for assignment.

  • No spaces around = for default parameter values (keywords).

  • Spaces around mathematical operators, but group them sensibly.

  • No multiple statements on the same line.

  • Naming conventions:

    Function names, variable names, and filenames should be descriptive and self explanatory. Avoid using abbreviations that are ambiguous or unfamiliar to readers outside your project, and do not abbreviate by deleting letters within a word. Avoid single letter variables if possible and use more verbose names for clarity. An exception for this are indexes in loops (i, j, k, etc).

    The following table summarizes the conventions:

    Type Public Internal
    Packages lower_with_under
    Modules lower_with_under _lower_with_under
    Classes CapWords _CapWords
    Exceptions CapWords
    Functions lower_with_under() _lower_with_under()
    Global/Class Constants CAPS_WITH_UNDER _CAPS_WITH_UNDER
    Global/Class Variables lower_with_under _lower_with_under
    Instance Variables lower_with_under _lower_with_under (protected)
    Method Names lower_with_under() _lower_with_under() (protected)
    Function/Method Parameters lower_with_under
    Local Variables lower_with_under

    Source: Google’s python style guide

  • Create an ignored variable:

    If you need to assign something (for instance, in Unpacking) but will not need that variable, use __ (double underscore):

    precip, __, metadata = import_bom_rf3('example_file.bom')

    Many Python style guides recommend the use of a single underscore “_” rather than the double underscore “__” recommended here. The issue is that “_” is commonly used as an alias for the gettext() function, and is also used at the interactive prompt to hold the value of the last operation. Using a double underscore instead is just as clear and eliminates the risk of accidentally interfering with either of these other use cases. (Source: https://docs.python-guide.org/writing/style/)

  • Zen of Python (PEP 20), the guiding principles for Python’s design:

    >>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

For a detailed description of a pythonic code style check these guidelines:

Auto-formatters

Formatting code to PEP8 style is a time consuming process. Instead of manually formatting code before a commit to to PEP8 style, you can use auto-format packages which automatically formats Python code to conform to the PEP 8 style guide.

If your development environment does not include auto-formatting capabilities, we recommend using black, which can be installed by any of the following options:

conda install black

#For the latest version:
conda install -c conda-forge black

pip install black

Check the official documentation for more information.

Docstrings

Every module, function, or class must have a docstring that describe its purpose and how to use it, following the conventions described in the PEP 257 and the Numpy’s docstrings format.

Here is a summary of the most important rules:

  • One-line docstrings Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.
  • A one-line docstring is a phrase ending in a period.
  • All docstrings should be written in imperative (“”“Return some value.”“”) mood rather than descriptive mood (“”“Returns some value.”“”).

Here is an example of a docstring:

def adjust_lag2_corrcoef1(gamma_1, gamma_2):
    """A simple adjustment of lag-2 temporal autocorrelation coefficient to
    ensure that the resulting AR(2) process is stationary when the parameters
    are estimated from the Yule-Walker equations.

    Parameters
    ----------

    gamma_1 : float
      Lag-1 temporal autocorrelation coeffient.

    gamma_2 : float
      Lag-2 temporal autocorrelation coeffient.

    Returns
    -------

    out : float
      The adjusted lag-2 correlation coefficient.
    """
Working on changes

IMPORTANT

If your changes will take a significant amount of work, we highly recommend opening an issue first, explaining what do you want to do and why. It is better to start the discussions early in case that other contributors disagree with what you would like to do or have ideas that will help you do it.

Collaborators guidelines

As a collaborator, all the new contributions that you want should be done in a new branch under your forked repository. Working on the master branch is reserved for Core Contributors only. Core Contributors are developers that actively work and maintain the repository. They are the only ones who accept pull requests and push commits directly to the pysteps repository.

To include the contributions for collaborators, we use the usual GitHub pull-request flow. In their simplest form, pull requests are a mechanism for a collaborator to notify to the pysteps project about a completed feature”.

Once your proposed changes are ready, you need to create a pull request via your GitHub account. Afterward, the core developers review the code and merge it into the master branch. Be aware that pull requests are more than just a notification, they are also an excellent place for discussing the proposed feature. If there is any problem with the changes, the other project collaborators can post feedback and the author of the commit can even fix the problems by pushing follow-up commits to feature branch.

Do not squash your commits after you have submitted a pull request, as this erases context during the review. The commits will be squashed when the pull request is merged.

To keep you forked repository clean, we suggest deleting branches for once the Pull Requests (PRs) are accepted and merged.

Once you’ve created a pull request, you can push commits from your topic branch to add them to your existing pull request. These commits will appear in chronological order within your pull request and the changes will be visible in the “Files changed” tab.

Other contributors can review your proposed changes, add review comments, contribute to the pull request discussion, and even add commits to the pull request.

Important: each PR should should only address a single objective (e.g. fix a bug, improve documentation, etc). Pushing changes to an open PR that are outside its objective are highly discouraged. Under this circumstances, the recommended way to proceed is creating a new PR for changes, clearly explaining their goal.

Testing your changes

Before committing changes or creating pull requests, check that the all the tests in the pysteps suite pass. See the Testing pySTEPS for the instruction to run the tests.

Although it is not strictly needed, we suggest creating minimal tests for new contributions to ensure that it achieves the desired behavior. Pysteps uses the pytest framework, that it is easy to use and also supports complex functional testing for applications and libraries. Check the pytests official documentation for more information.

The tests should be placed under the pysteps.tests module. The file should follow the test_*.py naming convention and have a descriptive name.

A quick way to get familiar with the pytest syntax and the testing procedures is checking the python scripts present in the pysteps test module.

Core developer guidelines

Working directly on the master branch is discouraged and is reserved only for small changes and updates that do not compromise the stability of the code. The master branch is a production branch that is ready to be deployed (cloned, installed, and ready to use). In consequence, this master branch is meant to be stable.

The pysteps repository uses a Travis CI, a Continuous Integration service that automatically runs a series of tests every time you commit to GitHub. In that way, your modifications along with the entire package is tested.

Pushing untested or work-in-progress changes to the master branch can potentially introduce bugs or brake the stability of the package. Since the tests takes around 10 minutes and the are run after the commit was pushed, any errors introduced in that commit will be noticed after the stable in the master branch was compromised. In addition, other developers start working on a new feature from master, they may start a potentially broken state.

Instead, it is recommended to work on each new feature in its own branch, which can be pushed to the central repository for backup/collaboration. When you’re done with the development work on the feature, then you can merge the feature branch into the master or submit a Pull Request. This approach has two main advantages:

  • Every commit on the feature branch is tested using Travis CI. If the tests fail, they do not affect the master branch.
  • Once the new feature, improvement, or bug correction is finished and the all tests passed, the commits history can be squashed into a single commit and then merged into the master branch.

This helps approach helps to keep the commits history clean and allows experimentation in the branch without compromising the stability of the package.

Processing pull requests

Core developers should follow these rules when processing pull requests:

  • Always wait for tests to pass before merging PRs.

  • Use “Squash and merge” to merge PRs.

  • Delete branches for merged PRs (by core devs pushing to the main repo).

  • Edit the final commit message before merging to conform to the following style to help having a clean git log output:

    • When merging a multi-commit PR make sure that the commit message doesn’t contain the local history from the committer and the review history from the PR. Edit the message to only describe the end state of the PR.
    • Make sure there is a single newline at the end of the commit message. This way there is a single empty line between commits in git log output.
    • Split lines as needed so that the maximum line length of the commit message is under 80 characters, including the subject line.
    • Capitalize the subject and each paragraph.
    • Make sure that the subject of the commit message has no trailing dot.
    • Use the imperative mood in the subject line (e.g. “Fix typo in README”).
    • If the PR fixes an issue, make sure something like “Fixes #xxx.” occurs in the body of the message (not in the subject).
Preparing a new release

Core developers should follow the steps to prepare a new release (version):

  1. Before creating the actual release in GitHub, be sure that every item in the following checklist was followed:

    • In the file setup.py, update the version=”X.X.X” keyword in the setup function.
    • Update the version in PKG-INFO file.
    • If new dependencies were added to pysteps since the last release, add them to the environment.yml, requirements.txt, and requirements_dev.txt files.

  2. Create a new release in GitHub following these guidelines. Include a detailed changelog in the release.

  3. Generating the source distribution for new pysteps version and upload it to the Python Package Index (PyPI). See Packaging the pysteps project for a detailed description of this process.

  4. Update the conda-forge pysteps-feedstock following this guidelines: Updating the conda-forge pysteps-feedstock

Credits

This documents was based in contributors guides of two Python open source projects:

Testing pySTEPS

The pysteps distribution includes a small test suite for some of the modules. To run the tests the pytest package is needed. To install it, in a terminal run:

pip install pytest
Automatic testing

The simplest way to run the pysteps’ test suite is using tox and the tox-conda plugin (conda needed). To install these packages activate your conda development environment and run:

conda install -c conda-forge tox tox-conda

Then, to run the tests, from the repo’s root run:

tox             # Run pytests
tox -e install  # Test package installation
tox -e black    # Test for black formatting warnings
Manual testing
Example data

The pySTEPS build-in tests require the pySTEPS example data installed. See the installation instructions in the Installing the Example data section.

Test an installed package

After the package is installed, you can launch the test suite from any directory by running:

pytest --pyargs pysteps
Test from sources

Before testing the package directly from the sources, we need to build the extensions in-place. To do that, from the root pysteps folder run:

python setup.py build_ext -i

Now, the package sources can be tested in-place using the pytest command on the root of the pysteps source directory. E.g.:

pytest -v --tb=line

Building the docs

The pysteps documentations is build using Sphinx, a tool that makes it easy to create intelligent and beautiful documentation

The documentation is located in the doc folder in the pysteps repo.

Automatic build

The simplest way to build the documentation is using tox and the tox-conda plugin (conda needed). To install these packages activate your conda development environment and run:

conda install -c conda-forge tox tox-conda

Then, to build the documentation, from the repo’s root run:

`tox -e docs`

This will create a conda environment will all the necessary dependencies and the data needed to create the examples.

Manual build

To build the docs you need to need to satisfy a few more dependencies related to Sphinx that are specified in the doc/requirements.txt file:

  • sphinx
  • numpydoc
  • sphinxcontrib.bibtex
  • sphinx_rtd_theme
  • sphinx_gallery

You can install these packages running pip install -r doc/requirements.txt.

In addition to this requirements, to build the example gallery in the documentation the example pysteps-data is needed. To download and install this data see the installation instructions in the Installing the Example data section.

Once these requirements are met, to build the documentation, in the doc folder run:

make html

This will build the documentation along with the example gallery.

The build documentation (html web page) will be available in doc/_build/html/. To correctly visualize the documentation, you need to set up and run a local HTTP server. To do that, in the doc/_build/html/ directory run:

python -m http.server

This will set up a local HTTP server on 0.0.0.0 port 8000. To see the built documentation open the following url in the browser: http://0.0.0.0:8000/

Packaging the pysteps project

The Python Package Index (PyPI) is a software repository for the Python programming language. PyPI helps you find and install software developed and shared by the Python community.

The following guide to package pysteps was adapted from the PyPI official documentation.

Generating the source distribution

The first step is to generate a source distribution (sdist) for the pysteps library. These are archives that are uploaded to the Package Index and can be installed by pip.

To create the sdist package we need the setuptools package installed.

Then, from the root folder of the pysteps source run:

python setup.py sdist

Once this command is completed, it should generate a tar.gz (source archive) file the dist directory:

dist/
  pysteps-a.b.c.tar.gz

where a.b.c denote the version number.

Uploading the source distribution to the archive

The last step is to upload your package to the Python Package Index.

Important

Before we actually upload the distribution to the Python Index, we will test it in Test PyPI. Test PyPI is a separate instance of the package index that allows us to try the distribution without affecting the real index (PyPi). Because TestPyPI has a separate database from the actual PyPI, you’ll need a separate user account for specifically for TestPyPI. You can register your account in https://test.pypi.org/account/register/.

Once you are registered, you can use twine to upload the distribution packages. Alternatively, the package can be uploaded manually from the Test PyPI page.

If Twine is not installed, you can install it by running pip install twine or conda install twine.

Test PyPI

To upload the recently created source distribution (dist/pysteps-a.b.c.tar.gz) under the dist directory run:

twine upload --repository-url https://test.pypi.org/legacy/ dist/pysteps-a.b.c.tar.gz

where a.b.c denote the version number.

You will be prompted for the username and password you registered with Test PyPI. After the command completes, you should see output similar to this:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading pysteps-a.b.c.tar.gz
100%|█████████████████████| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

Once uploaded your package should be viewable on TestPyPI, for example, https://test.pypi.org/project/pysteps

Test the uploaded package

Before uploading the package to the official Python Package Index, test that the package can be installed using pip.

Automatic test

The simplest way to hat the package can be installed using pip is using tox and the tox-conda plugin (conda needed). To install these packages activate your conda development environment and run:

conda install -c conda-forge tox tox-conda

Then, to test the installation in a minimal and an environment with all the dependencies (full env), run:

tox -r -e pypi_test        # Test the installation in a minimal env
tox -r -e pypi_test_full   # Test the installation in an full env
Manual test

To manually test the installation on new environment, create a copy of the basic development environment using the environment_dev.yml file in the root folder of the pysteps project:

conda create -f environment_dev.yml -n pysteps_test

Then we activate the environment:

source activate pysteps_test

or:

conda activate pysteps_test

If the environment pysteps_test was already created, remove any version of pysteps already installed:

pip uninstall pysteps

Now, install the pysteps package from test.pypi.org:

pip install --index-url https://test.pypi.org/simple/ pysteps

To test that the installation was successful, from a folder different than the pysteps source, run:

pytest --pyargs pysteps

If any test didn’t pass, check the sources or consider creating a new release fixing those bugs.

Upload package to PyPi

Once the sdist package was tested, we can safely upload it to the Official PyPi repository with:

twine upload dist/pysteps-a.b.c.tar.gz

Now, pysteps can be installed by simply running:

pip install pysteps

As an extra sanity measure, it is recommended to test the pysteps package installed from the Official PyPi repository (instead of the test PyPi).

Automatic test

Similarly to the Test the uploaded package section, to test the installation from PyPI in a clean environment, run:

tox -r -e pypi
Manual test

Follow test instructions in Test PyPI section.

Updating the conda-forge pysteps-feedstock

Here we will describe the steps to update the pysteps conda-forge feedstock. This tutorial is intended for the core developers listed as maintainers of the conda recipe in the conda-forge/pysteps-feedstock.

Examples for needing to update the pysteps-feedstock are:

  • New release
  • Fix errors pysteps package errors

The following tutorial was adapted from the official conda-forge.org documentation, released under CC4.0 license

What is a “conda-forge”

Conda-forge is a community effort that provides conda packages for a wide range of software. The conda team from Anaconda packages a multitude of packages and provides them to all users free of charge in their default channel.

conda-forge is a community-led conda channel of installable packages that allows users to share software that is not included in the official Anaconda repository. The main advantages of conda-forge are:

  • all packages are shared in a single channel named conda-forge
  • care is taken that all packages are up-to-date
  • common standards ensure that all packages have compatible versions
  • by default, packages are built for macOS, linux amd64 and windows amd64

In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

The actual pysteps feedstock is https://github.com/conda-forge/pysteps-feedstock

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services.

See the official conda-forge documentation for more details.

Maintain pysteps conda-forge package

Pysteps Core developers that are maintainers of the pysteps feedstock.

All pysteps developers listed as maintainers of the pysteps feedstock are given push access to the feedstock repository. This means that a maintainer can create branches in the main repository.

Every time that a new commit is pushed/merged in the feedstock repository, conda-forge runs Continuous Integration (CI) system that run quality checks, builds the pysteps recipe on Windows, OSX, and Linux, and publish the built recipes in the conda-forge channel.

Important

For updates, using a branch in the main repo and a subsequent Pull Request (PR) to the master branch is discouraged because: - CI is run on both the branch and on the Pull Request (if any) associated with that branch. This wastes CI resources. - Branches are automatically published by the CI system. This mean that a for every push, the packages will be published before the PR is actually merged.

For these reasons, to update the feedstock, the maintainers need to fork the feedstock, create a new branch in that fork, push to that branch in the fork, and then open a PR to the conda-forge repo.

Workflow for updating a pysteps-feedstock

The mandatory steps to update the pysteps-feedstock are:

  1. Forking the pysteps-feedstock.

    • Clone the forked repository in your computer:

      git clone https://github.com/<your-github-id>/pysteps-feedstock

  2. Syncing your fork with the pysteps feedstock. This step is only needed if your local repository is not up to date the pysteps-feedstock. If you just cloned the forked pysteps-feedstock, you can ignore this step.

    • Make sure you are on the master branch:

      git checkout master

    • Register conda-forge’s feedstock with:

      git remote add upstream https://github.com/conda-forge/pysteps-feedstock

    • Fetch the latest updates with git fetch upstream:

      git fetch upstream

    • Pull in the latest changes into your master branch:

      git rebase upstream/master

  3. Create a new branch:

    git checkout -b <branch-name>

  4. Update the recipe and push changes in this new branch

    • See next section “Updating recipes” for more details

    • Push changes:

      git commit -m <commit message>

  5. Pushing your changes to GitHub:

    git push origin <branch-name>

  6. Propose a Pull Request

    • Create a pull request via the web interface
Updating pysteps recipe

The pysteps-feedstock should be updated when:

  • We release a new pysteps version
  • Need to fix errors in the pysteps package
New release

When a new pysteps version is released, before update the pysteps feedstock, the new version needs to be uploaded to the Python Package Index (PyPI) (see Packaging the pysteps project for more details). This step is needed because the conda recipe uses the PyPI to build the pysteps conda package.

Once the new version is available in the PyPI, the conda recipe in pysteps-feedstock/recipe/meta.yaml needs to be updated by:

  1. Updating version and hash
  2. Checking the dependencies
  3. When the package version changes, reset the build number back to 0.

The build number is increased when the source code for the package has not changed but you need to make a new build. As a rule of thumb, the build number is increased whenever a new package with the same version needs to be uploaded to the conda-forge channel.

Recipe fixing

In case that the recipe must be updated but the source code for the package has not changed the build_number in the conda recipe in pysteps-feedstock/recipe/meta.yaml needs to be increased by 1.

Some examples for needing to increase the build number are:

  • updating the pinned dependencies
  • Fixing wrong dependencies

Bibliography

[BPS06]N. E. Bowler, C. E. Pierce, and A. W. Seed. STEPS: a probabilistic precipitation forecasting scheme which merges an extrapolation nowcast with downscaled NWP. Quarterly Journal of the Royal Meteorological Society, 132(620):2127–2155, 2006. doi:10.1256/qj.04.100.
[BrockerS07]J. Bröcker and L. A. Smith. Increasing the reliability of reliability diagrams. Weather and Forecasting, 22(3):651–661, 2007. doi:10.1175/WAF993.1.
[CRS04]B. Casati, G. Ross, and D. B. Stephenson. A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteorological Applications, 11(2):141––154, 2004. doi:10.1017/S1350482704001239.
[EWW+13]E. Ebert, L. Wilson, A. Weigel, M. Mittermaier, P. Nurmi, P. Gill, M. Göber, S. Joslyn, B. Brown, T. Fowler, and A. Watkins. Progress and challenges in forecast verification. Meteorological Applications, 20(2):130–139, 2013. doi:10.1002/met.1392.
[GZ02]U. Germann and I. Zawadzki. Scale-dependence of the predictability of precipitation from continental radar images. Part I: description of the methodology. Monthly Weather Review, 130(12):2859–2873, 2002. doi:10.1175/1520-0493(2002)130<2859:SDOTPO>2.0.CO;2.
[Her00]H. Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15(5):559–570, 2000. doi:10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2.
[NBS+17]D. Nerini, N. Besic, I. Sideris, U. Germann, and L. Foresti. A non-stationary stochastic ensemble generator for radar rainfall fields based on the short-space Fourier transform. Hydrology and Earth System Sciences, 21(6):2777–2797, 2017. doi:10.5194/hess-21-2777-2017.
[PvGPO94]M. Proesmans, L. van Gool, E. Pauwels, and A. Oosterlinck. Determination of optical flow and its discontinuities using non-linear diffusion. In J.-O. Eklundh, editor, Computer Vision — ECCV ‘94, volume 801 of Lecture Notes in Computer Science, pages 294–304. Springer Berlin Heidelberg, 1994.
[PCH18]S. Pulkkinen, V. Chandrasekar, and A.-M. Harri. Nowcasting of precipitation in the high-resolution Dallas-Fort Worth (DFW) urban radar remote sensing network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(8):2773–2787, 2018. doi:10.1109/JSTARS.2018.2840491.
[RL08]N. M. Roberts and H. W. Lean. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Monthly Weather Review, 136(1):78–97, 2008. doi:10.1175/2007MWR2123.1.
[RC11]E. Ruzanski and V. Chandrasekar. Scale filtering for improved nowcasting performance in a high-resolution X-band radar network. IEEE Transactions on Geoscience and Remote Sensing, 49(6):2296–2307, June 2011.
[RCW11]E. Ruzanski, V. Chandrasekar, and Y. Wang. The CASA nowcasting system. Journal of Atmospheric and Oceanic Technology, 28(5):640–655, 2011. doi:10.1175/2011JTECHA1496.1.
[See03]A. W. Seed. A dynamic and spatial scaling approach to advection forecasting. Journal of Applied Meteorology, 42(3):381–388, 2003. doi:10.1175/1520-0450(2003)042<0381:ADASSA>2.0.CO;2.
[SPN13]A. W. Seed, C. E. Pierce, and K. Norman. Formulation and evaluation of a scale decomposition-based stochastic precipitation nowcast scheme. Water Resources Research, 49(10):6624–6641, 2013. doi:10.1002/wrcr.20536.
[ZR09]P. Zacharov and D. Rezacova. Using the fractions skill score to assess the relationship between an ensemble QPF spread and skill. Atmospheric Research, 94(4):684–693, 2009. doi:10.1016/j.atmosres.2009.03.004.