IBF-river-flood-pipeline

Forecast riverine flooding. Part of IBF-system.

Description

The pipeline roughly consists of three steps:

• Extract data on river discharge from an external provider, both in specific locations (stations) and over pre-defined areas (administrative divisions).
• Forecast floods by determining if the river discharge is higher than pre-defined thresholds; if so, calculate flood extent and impact (affected people).
• Send this data to the IBF app.

The pipeline stores data in:

• ibf-cosmos (Azure Cosmos DB): river discharge per station / administrative division, flood forecasts, and trigger thresholds
• 510ibfsystem (Azure Storage Account): raw data from GloFAS and other geospatial data

The pipeline depends on the following services:

• GloFAS: provides river discharge forecasts
• Glofas data pipeline in IBF-data-factory (Azure Data Factory): extracts GloFAS data and stores it in 510ibfsystem
• IBF-app

For more information, see the functional architecture diagram.

Basic Usage

To run the pipeline locally

1. fill in the secrets in .env.example and rename the file to .env; in this way, they will be loaded as environment variables
2. install requirements

pip install poetry
poetry install --no-interaction

3. run the pipeline with python flood_pipeline.py

Usage: flood_pipeline.py [OPTIONS]

Options:
  --country TEXT  country ISO3
  --prepare       prepare discharge data
  --extract       extract discharge data
  --forecast      forecast floods
  --send          send to IBF app
  --save          save to storage
  --help          Show this message and exit.

Advanced Usage

How do I set up the pipeline for a new country?

1. Check that the administrative boundaries are in the IBF system; if not, ask IBF developers to add them
2. Add country-specific configuration in config/config.yaml
3. Create historical flood extent maps

python data_updates\add_flood_maps.py --country <country ISO3>

3. Compute trigger and alert thresholds

python data_updates\add_flood_thresholds.py --country <country ISO3>

4. Update Glofas data pipeline in IBF-data-factory so that it will trigger a pipeline run for the new country

How do I insert an exception for a specific country?

You don't. The pipeline is designed to work in the same way for all countries. If you need to change the pipeline's behavior for a specific country, please discuss your needs with your fellow data specialist, they will try their best to accommodate your request.

There is a new version of GloFAS, how do I update the pipeline?

GloFAS should take care to update the river discharge data in a backward-compatible way. If that is not the case, you need to have a look at floodpipeline/extract.py and change what's needed.

What will probably change with the new GloFAS version are the trigger/alert thresholds. To update them you need to

1. Change the