Model inter-comparison study

This wiki was initially created as part of the Water Research Commission (WRC) project, Critical catchment model inter-comparison and model use guidance development (K5-2927), which ran from 2019 to 2021. The project team members are listed here.

This project:

• compared the model structure options and capabilities of different commonly used modelling software tools
• modelled four different case study catchments, including scenarios of change, using this set of tools to explore practical implications of their differences

Project Abstract

Catchment hydrological modelling has become a central component of water resources management in South Africa. As such, there is need for continuous research and capacity building to enable the water sector to take advantage of, and make wise-use of, the diversity of modelling strategies and tools available. To support this, the Critical catchment hydrological model inter-comparison and model use guidance development (K5-2927) project explored the structural options across several commonly used tools: WRSM-Pitman, SPATSIM-Pitman, ACRU4, SWAT2012, and MIKE-SHE. This included initial assessment of the potential impacts of tool differences in common use cases, done though applying all of them to a variety of case studies. A survey of the modelling community confirmed which tools are being used and looked at user experiences. The project highlighted issues needing attention, particularly use of uncertainty and realism assessments regardless of the tools used.

The tools showed high-level similarities in basic capabilities, however there were many notable differences in model spatial units and subsurface layers, flows between layers and units, the scale at which climate inputs are specified, algorithms that calculate flows, and time steps used. These result in differences in what can be explicitly modelled in a given tool. Tool interfaces also differ significantly, influencing set-up efficiency and transparency, the ease with which the range of reasonable parameter values can be explored for calibration and uncertainty analyses, and the ease with which different water balance outputs can be obtained to assess model realism. Each tool had some type of advantage over others. To assist users in weighting what is important for their use cases, tool capabilities and options were summarised side-by-side on a modelling guidance ‘wiki’ website created in the project. A key finding of the case study modelling exercise was that, while models reaching acceptable performance against observed streamflow data could be built with any of the tools, these models could be predicting quite different balances of processes. This was evident when extracting detailed water balance outputs. Differences in baseline process representation resulted in differing predicted yield changes when the models were applied to scenarios (by up to 20% in the most extreme example). This highlights the need to assess model water balances and find ways to address the challenges in doing so: some tool interfaces make this very time consuming and there is often a lack of auxiliary data for comparison.

Follow-on project on modelling uncertainty and the "model-a-thon" activity

The model inter-comparison study highlighted the fact that there can be many justifiable ways of setting up a model of a given catchment with a given dataset, even within a single modelling tool, the modeller must make many subjective decisions. The case study modelling also demonstrated that models with different structures can be applied to the same catchment area and achieve satisfactory calibration (can recreate observed streamflow reasonably well), but then the models can produce very different predictions of the hydrological impacts of a change scenario. These differences arose because the various models were representing the mix of internal catchment processes, storages and fluxes differently (e.g. canopy interception evaporation, evapotranspiration from soil and groundwater, surface runoff generation, soil water storage and interflow, groundwater storage and outflow). If the speeds of various flow pathways and their connections differ across models, different mixtures of these can produce similar baseline streamflow predictions to one another in calibration. They reach the same answer for different reasons. However these differences become salient when we change things to represent an alternative scenario, such as different climate or land cover, and then the predictions across these differently structured models diverge. If we cannot actually tell which of these model representations of the catchment is more realistic, then the range of predictions across all of these models is an indication of the uncertainty in prediction of the scenario impact. This is uncertainty due to uncertainty about the model's structure. There will also be uncertainty in predictions due to uncertainties in the input data and parameter values used.

This led the project team