Forecast skill is always evaluated against a reference for verification. In ESM-based predictions, it is common to compare the ensemble mean forecast against the reference.
In hindcast ensembles
climpred.prediction.compute_hindcast(), this ensemble mean forecast (
comparison='e2r') is expected to perform better than individual ensemble members (
comparison='m2r') as the chaotic component of forecasts is expected to be suppressed by this averaging, while the memory of the system sustains. [Boer2016]
HindcastEnsemble skill is computed by default as the ensemble mean forecast against the reference (
In perfect-model frameworks
climpred.prediction.compute_perfect_model(), there are even more ways of comparisons. [Seferian2018] shows comparison of the ensemble members against the control run (
comparison='m2c') and ensemble members against all other ensemble members (
comparison='m2m'). Furthermore, using the ensemble mean forecast can be also verified against one control member (
comparison='e2c') or all members (
comparison='m2e') as done in [Griffies1997].
Perfect-model framework comparison defaults to the ensemble mean forecast verified against each member in turns (
These different comparisons demand for a normalization factor to arrive at a normalized skill of 1, when skill saturation is reached (ref: metrics).
While HindcastEnsemble skill is computed over all initializations
init of the hindcast, the resulting skill is a mean forecast skill over all initializations.
PerfectModelEnsemble skill is computed over a supervector comprised of all initializations and members, which allows the computation of the ACC-based skill [Bushuk2018], but also returns a mean forecast skill over all initializations.
The supervector approach shown in [Bushuk2018] and just calculating a distance-based metric like
rmse over the member dimension as in [Griffies1997] yield very similar results.
_e2r(ds, reference[, stack_dims])
|Compare the ensemble mean forecast to a reference in HindcastEnsemble.
_m2r(ds, reference[, stack_dims])
|Compares each member individually to a reference in HindcastEnsemble.
_m2e(ds[, supervector_dim, stack_dims])
|Create two supervectors to compare all members to ensemble mean while
_m2c(ds[, supervector_dim, control_member, …])
|Create two supervectors to compare all members to control.
_m2m(ds[, supervector_dim, stack_dims])
|Create two supervectors to compare all members to all others in turn.
_e2c(ds[, supervector_dim, control_member, …])
|Create two supervectors to compare ensemble mean to control.
|[Boer2016]||Boer, G. J., D. M. Smith, C. Cassou, F. Doblas-Reyes, G. Danabasoglu, B. Kirtman, Y. Kushnir, et al. “The Decadal Climate Prediction Project (DCPP) Contribution to CMIP6.” Geosci. Model Dev. 9, no. 10 (October 25, 2016): 3751–77. https://doi.org/10/f89qdf.|
|[Bushuk2018]||(1, 2) Mitchell Bushuk, Rym Msadek, Michael Winton, Gabriel Vecchi, Xiaosong Yang, Anthony Rosati, and Rich Gudgel. Regional Arctic sea–ice prediction: potential versus operational seasonal forecast skill. Climate Dynamics, June 2018. https://doi.org/10/gd7hfq.|
|[Griffies1997]||(1, 2) |
- Griffies and K. Bryan. A predictability study of simulated North Atlantic multidecadal variability. Climate Dynamics, 13(7-8):459–487, August 1997. https://doi.org/10/ch4kc4.
|[Seferian2018]||Roland Séférian, Sarah Berthet, and Matthieu Chevallier. Assessing the Decadal Predictability of Land and Ocean Carbon Uptake. Geophysical Research Letters, March 2018. https://doi.org/10/gdb424.|