References used in the documentation and used for studying predictability.


Emily Becker, Huug van den Dool, and Qin Zhang. Predictability and Forecast Skill in NMME. Journal of Climate, 27(15):5891–5906, August 2014. doi:10.1175/JCLI-D-13-00597.1.


Yoav Benjamini and Yosef Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):13, 1994. doi:10.1111/j.2517-6161.1995.tb02031.x.


G. J. Boer. Long time-scale potential predictability in an ensemble of coupled climate models. Climate Dynamics, 23(1):29–44, August 2004. doi:10/csjjbh.


G. J. Boer, D. M. Smith, C. Cassou, F. Doblas-Reyes, G. Danabasoglu, B. Kirtman, Y. Kushnir, M. Kimoto, G. A. Meehl, R. Msadek, W. A. Mueller, K. E. Taylor, F. Zwiers, M. Rixen, Y. Ruprich-Robert, and R. Eade. The Decadal Climate Prediction Project (DCPP) contribution to CMIP6. Geosci. Model Dev., 9(10):3751–3777, October 2016. doi:10/f89qdf.


Riley X. Brady, Nicole S. Lovenduski, Stephen G. Yeager, Matthew C. Long, and Keith Lindsay. Skillful multiyear predictions of ocean acidification in the California Current System. Nature Communications, 11(1):1–9, May 2020. doi:10/ggtpks.


Riley X. Brady and Aaron Spring. Climpred: Verification of weather and climate forecasts. Journal of Open Source Software, 6(59):2781, March 2021. doi:10/gh9646.


Grant Branstator and Haiyan Teng. Two Limits of Initial-Value Decadal Predictability in a CGCM. Journal of Climate, 23(23):6292–6311, August 2010. doi:10/bwq92h.


Christopher S. Bretherton, Martin Widmann, Valentin P. Dymnikov, John M. Wallace, and Ileana Bladé. The Effective Number of Spatial Degrees of Freedom of a Time-Varying Field. Journal of Climate, 12(7):1990–2009, July 1999. doi:10/bs7fv4.


Glenn W. Brier. Verficiation of forecasts expressed int terms of probability. Monthly Weather Review, 1950. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.


Mitchell Bushuk, Rym Msadek, Michael Winton, Gabriel Vecchi, Xiaosong Yang, Anthony Rosati, and Rich Gudgel. Regional Arctic sea–ice prediction: potential versus operational seasonal forecast skill. Climate Dynamics, June 2018. doi:10/gd7hfq.


M. Collins. Climate predictability on interannual to decadal time scales: the initial value problem. Climate Dynamics, 19(8):671–692, October 2002. doi:10/czfb45.


Timothy DelSole and Michael K. Tippett. Forecast Comparison Based on Random Walks. Monthly Weather Review, 144(2):615–626, February 2016. doi:10/f782pf.


X. Feng, T. DelSole, and P. Houser. Bootstrap estimated seasonal potential predictability of global temperature and precipitation. Geophysical Research Letters, 2011. doi:10/ft272w.


Filippa Fransner, François Counillon, Ingo Bethke, Jerry Tjiputra, Annette Samuelsen, Aleksi Nummelin, and Are Olsen. Ocean Biogeochemical Predictions—Initialization and Limits of Predictability. Frontiers in Marine Science, 2020. doi:10/gg22rr.


Thomas L. Frölicher, Luca Ramseyer, Christoph C. Raible, Keith B. Rodgers, and John Dunne. Potential predictability of marine ecosystem drivers. Biogeosciences, 17(7):2061–2083, April 2020. doi:10/ggr9mb.


Tilmann Gneiting and Adrian E Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359–378, March 2007. doi:10/c6758w.


L. Goddard, A. Kumar, A. Solomon, D. Smith, G. Boer, P. Gonzalez, V. Kharin, W. Merryfield, C. Deser, S. J. Mason, B. P. Kirtman, R. Msadek, R. Sutton, E. Hawkins, T. Fricker, G. Hegerl, C. a. T. Ferro, D. B. Stephenson, G. A. Meehl, T. Stockdale, R. Burgman, A. M. Greene, Y. Kushnir, M. Newman, J. Carton, I. Fukumori, and T. Delworth. A verification framework for interannual-to-decadal predictions experiments. Climate Dynamics, 40(1-2):245–272, January 2013. doi:10/f4jjvf.


J. Gottschalck, M. Wheeler, K. Weickmann, F. Vitart, N. Savage, H. Lin, H. Hendon, D. Waliser, K. Sperber, M. Nakagawa, C. Prestrelo, M. Flatau, and W. Higgins. A Framework for Assessing Operational Madden–Julian Oscillation Forecasts: A CLIVAR MJO Working Group Project. Bulletin of the American Meteorological Society, 91(9):1247–1258, September 2010. doi:10/fcftn7.


S. M. Griffies and K. Bryan. A predictability study of simulated North Atlantic multidecadal variability. Climate Dynamics, 13(7-8):459–487, August 1997. doi:10/ch4kc4.


Thomas M. Hamill. Interpretation of Rank Histograms for Verifying Ensemble Forecasts. Monthly Weather Review, 129(3):550–560, March 2001. doi:10/dkkvh3.


Ed Hawkins, Buwen Dong, Jon Robson, Rowan Sutton, and Doug Smith. The Interpretation and Use of Biases in Decadal Climate Predictions. Journal of Climate, 27(8):2931–2947, January 2014. doi:10/f5w2xr.


Ed Hawkins, Steffen Tietsche, Jonathan J. Day, Nathanael Melia, Keith Haines, and Sarah Keeley. Aspects of designing and evaluating seasonal-to-interannual Arctic sea-ice prediction systems. Quarterly Journal of the Royal Meteorological Society, 142(695):672–683, January 2016. doi:10/gfb3pn.


Sebastian Illing, Christopher Kadow, Kunst Oliver, and Ulrich Cubasch. MurCSS: A Tool for Standardized Evaluation of Decadal Hindcast Systems. Journal of Open Research Software, 2(1):e24, September 2014. doi:10/gfxr7x.


Ian T. Jolliffe and David B. Stephenson. Forecast Verification: A Practitioner's Guide in Atmospheric Science. John Wiley & Sons, Ltd, Chichester, UK, December 2011. ISBN 978-1-119-96000-3 978-0-470-66071-3. doi:10.1002/9781119960003.


Christopher Kadow, Sebastian Illing, Oliver Kunst, Henning W. Rust, Holger Pohlmann, Wolfgang A. Müller, and Ulrich Cubasch. Evaluation of forecasts by accuracy and spread in the MiKlip decadal climate prediction system. Meteorologische Zeitschrift, pages 631–643, December 2016. doi:10/f9jrhw.


Ben Kirtman, Scott B. Power, Akintayo John Adedoyin, George J. Boer, Roxana Bojariu, Ines Camilloni, Francisco Doblas-Reyes, Arlene M. Fiore, Masahide Kimoto, Gerald Meehl, Michael Prather, Abdoulaye Sarr, Christoph Schär, Rowan Sutton, Geert Jan van Oldenborgh, Gabriel Vecchi, and Hui-Jun Wang. Near-term Climate Change: Projections and Predictability. In Thomas F. Stocker, D. Qin, G.-K. Plattner, M. Tignor, S. K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, and P.M. Midgley, editors, Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, 2013. URL: (visited on 2018-03-15).


Ben P. Kirtman, Dughong Min, Johnna M. Infanti, James L. Kinter, Daniel A. Paolino, Qin Zhang, Huug van den Dool, Suranjana Saha, Malaquias Pena Mendez, Emily Becker, Peitao Peng, Patrick Tripp, Jin Huang, David G. DeWitt, Michael K. Tippett, Anthony G. Barnston, Shuhua Li, Anthony Rosati, Siegfried D. Schubert, Michele Rienecker, Max Suarez, Zhao E. Li, Jelena Marshak, Young-Kwon Lim, Joseph Tribbia, Kathleen Pegion, William J. Merryfield, Bertrand Denis, and Eric F. Wood. The North American Multimodel Ensemble: Phase-1 Seasonal-to-Interannual Prediction; Phase-2 toward Developing Intraseasonal Prediction. Bulletin of the American Meteorological Society, 95(4):585–601, April 2014. doi:10/ggspp9.


K. M. Krumhardt, N. S. Lovenduski, M. C. Long, J. Y. Luo, K. Lindsay, S. Yeager, and C. Harrison. Potential Predictability of Net Primary Production in the Ocean. Global Biogeochemical Cycles, 34(6):e2020GB006531, 2020. doi:10/gg9ss8.


Hongmei Li, Tatiana Ilyina, Wolfgang A. Müller, and Frank Sienz. Decadal predictions of the North Atlantic CO2 uptake. Nature Communications, 7:11076, March 2016. doi:10/f8wkrs.


Nicolau Manubens, Louis-Philippe Caron, Alasdair Hunter, Omar Bellprat, Eleftheria Exarchou, Neven S. Fučkar, Javier Garcia-Serrano, François Massonnet, Martin Ménégoz, Valentina Sicardi, Lauriane Batté, Chloé Prodhomme, Verónica Torralba, Nicola Cortesi, Oriol Mula-Valls, Kim Serradell, Virginie Guemas, and Francisco J. Doblas-Reyes. An R package for climate forecast verification. Environmental Modelling & Software, 103:29–42, May 2018. doi:10/gc9wzk.


Jochem Marotzke, Wolfgang A. Müller, Freja S. E. Vamborg, Paul Becker, Ulrich Cubasch, Hendrik Feldmann, Frank Kaspar, Christoph Kottmeier, Camille Marini, Iuliia Polkova, Kerstin Prömmel, Henning W. Rust, Detlef Stammer, Uwe Ulbrich, Christopher Kadow, Armin Köhl, Jürgen Kröger, Tim Kruschke, Joaquim G. Pinto, Holger Pohlmann, Mark Reyers, Marc Schröder, Frank Sienz, Claudia Timmreck, and Markus Ziese. MiKlip: A National Research Project on Decadal Climate Prediction. Bulletin of the American Meteorological Society, 97(12):2379–2394, June 2016. doi:10/gddfck.


S. J. Mason. Understanding forecast verification statistics. Meteorological Applications, 15(1):31–40, March 2008. doi:10/bgvgnz.


James E. Matheson and Robert L. Winkler. Scoring Rules for Continuous Probability Distributions. Management Science, 22(10):1087–1096, June 1976. doi:10/cwwt4g.


Gerald A. Meehl, Lisa Goddard, George Boer, Robert Burgman, Grant Branstator, Christophe Cassou, Susanna Corti, Gokhan Danabasoglu, Francisco Doblas-Reyes, Ed Hawkins, Alicia Karspeck, Masahide Kimoto, Arun Kumar, Daniela Matei, Juliette Mignot, Rym Msadek, Antonio Navarra, Holger Pohlmann, Michele Rienecker, Tony Rosati, Edwin Schneider, Doug Smith, Rowan Sutton, Haiyan Teng, Geert Jan van Oldenborgh, Gabriel Vecchi, and Stephen Yeager. Decadal Climate Prediction: An Update from the Trenches. Bulletin of the American Meteorological Society, 95(2):243–267, April 2013. doi:10/f3cvw2.


Gerald A. Meehl, Lisa Goddard, James Murphy, Ronald J. Stouffer, George Boer, Gokhan Danabasoglu, Keith Dixon, Marco A. Giorgetta, Arthur M. Greene, Ed Hawkins, Gabriele Hegerl, David Karoly, Noel Keenlyside, Masahide Kimoto, Ben Kirtman, Antonio Navarra, Roger Pulwarty, Doug Smith, Detlef Stammer, and Timothy Stockdale. Decadal Prediction: Can It Be Skillful? Bulletin of the American Meteorological Society, 90(10):1467–1486, October 2009. doi:10/dpsjbp.


Gerald A. Meehl, Jadwiga H. Richter, Haiyan Teng, Antonietta Capotondi, Kim Cobb, Francisco Doblas-Reyes, Markus G. Donat, Matthew H. England, John C. Fyfe, Weiqing Han, Hyemi Kim, Ben P. Kirtman, Yochanan Kushnir, Nicole S. Lovenduski, Michael E. Mann, William J. Merryfield, Veronica Nieves, Kathy Pegion, Nan Rosenbloom, Sara C. Sanchez, Adam A. Scaife, Doug Smith, Aneesh C. Subramanian, Lantao Sun, Diane Thompson, Caroline C. Ummenhofer, and Shang-Ping Xie. Initialized Earth System prediction from subseasonal to decadal timescales. Nature Reviews Earth & Environment, pages 1–18, April 2021. doi:10/gjqkw6.


William J. Merryfield, Johanna Baehr, Lauriane Batté, Emily J. Becker, Amy H. Butler, Caio A. S. Coelho, Gokhan Danabasoglu, Paul A. Dirmeyer, Francisco J. Doblas-Reyes, Daniela I. V. Domeisen, Laura Ferranti, Tatiana Ilynia, Arun Kumar, Wolfgang A. Müller, Michel Rixen, Andrew W. Robertson, Doug M. Smith, Yuhei Takaya, Matthias Tuma, Frederic Vitart, Christopher J. White, Mariano S. Alvarez, Constantin Ardilouze, Hannah Attard, Cory Baggett, Magdalena A. Balmaseda, Asmerom F. Beraki, Partha S. Bhattacharjee, Roberto Bilbao, Felipe M. de Andrade, Michael J. DeFlorio, Leandro B. Díaz, Muhammad Azhar Ehsan, Georgios Fragkoulidis, Sam Grainger, Benjamin W. Green, Momme C. Hell, Johnna M. Infanti, Katharina Isensee, Takahito Kataoka, Ben P. Kirtman, Nicholas P. Klingaman, June-Yi Lee, Kirsten Mayer, Roseanna McKay, Jennifer V. Mecking, Douglas E. Miller, Nele Neddermann, Ching Ho Justin Ng, Albert Ossó, Klaus Pankatz, Simon Peatman, Kathy Pegion, Judith Perlwitz, G. Cristina Recalde-Coronel, Annika Reintges, Christoph Renkl, Balakrishnan Solaraju-Murali, Aaron Spring, Cristiana Stan, Y. Qiang Sun, Carly R. Tozer, Nicolas Vigaud, Steven Woolnough, and Stephen Yeager. Current and Emerging Developments in Subseasonal to Decadal Prediction. Bulletin of the American Meteorological Society, 101(6):869–896, June 2020. doi:10/ggvcqv.


Allan H Murphy and Richard W Katz. Probability, Statistics, and Decision Making in the Atmospheric Sciences. Westview Press, Boulder, CO, 1985. ISBN 9780367284336.


Allan H. Murphy. Skill Scores Based on the Mean Square Error and Their Relationships to the Correlation Coefficient. Monthly Weather Review, 116(12):2417–2424, December 1988. doi:10/fc7mxd.


Allan H. Murphy and Edward S. Epstein. Skill Scores and Correlation Coefficients in Model Verification. Monthly Weather Review, 117(3):572–582, March 1989. doi:10/fd6v47.


Kathy Pegion, Ben P. Kirtman, Emily Becker, Dan C. Collins, Emerson LaJoie, Robert Burgman, Ray Bell, Timothy DelSole, Dughong Min, Yuejian Zhu, Wei Li, Eric Sinsky, Hong Guan, Jon Gottschalck, E. Joseph Metzger, Neil P Barton, Deepthi Achuthavarier, Jelena Marshak, Randal D. Koster, Hai Lin, Normand Gagnon, Michael Bell, Michael K. Tippett, Andrew W. Robertson, Shan Sun, Stanley G. Benjamin, Benjamin W. Green, Rainer Bleck, and Hyemi Kim. The Subseasonal Experiment (SubX): A Multimodel Subseasonal Prediction Experiment. Bulletin of the American Meteorological Society, 100(10):2043–2060, July 2019. doi:10/ggkt9s.


Holger Pohlmann, Michael Botzet, Mojib Latif, Andreas Roesch, Martin Wild, and Peter Tschuck. Estimating the Decadal Predictability of a Coupled AOGCM. Journal of Climate, 17(22):4463–4472, November 2004. doi:10/d2qf62.


Harun A. Rashid, Harry H. Hendon, Matthew C. Wheeler, and Oscar Alves. Prediction of the Madden–Julian oscillation with the POAMA dynamical prediction system. Climate Dynamics, 36(3):649–661, February 2011. doi:10/b992k4.


L. Resplandy, R. Séférian, and L. Bopp. Natural variability of CO2 and O2 fluxes: What can we learn from centuries-long climate models simulations? Journal of Geophysical Research: Oceans, 120(1):384–404, January 2015. doi:10/f63c3h.


James S. Risbey, Dougal T. Squire, Amanda S. Black, Timothy DelSole, Chiara Lepore, Richard J. Matear, Didier P. Monselesan, Thomas S. Moore, Doug Richardson, Andrew Schepen, Michael K. Tippett, and Carly R. Tozer. Standard assessments of climate forecast skill can be misleading. Nature Communications, 12(1):4346, July 2021. doi:10/gk8k7k.


Doug M. Smith, Adam A. Scaife, George J. Boer, Mihaela Caian, Francisco J. Doblas-Reyes, Virginie Guemas, Ed Hawkins, Wilco Hazeleger, Leon Hermanson, Chun Kit Ho, Masayoshi Ishii, Viatcheslav Kharin, Masahide Kimoto, Ben Kirtman, Judith Lean, Daniela Matei, William J. Merryfield, Wolfgang A. Müller, Holger Pohlmann, Anthony Rosati, Bert Wouters, and Klaus Wyser. Real-time multi-model decadal climate predictions. Climate Dynamics, 41(11-12):2875–2888, December 2013. doi:10/f5g2rz.


Aaron Spring. Internal Variability and Potential Predictability of the Global Carbon Cycle in a Perfect-Model Framework. PhD thesis, Universität Hamburg Hamburg, June 2021. doi:10.17617/2.3289580.


Aaron Spring, István Dunkl, Hongmei Li, Victor Brovkin, and Tatiana Ilyina. Trivial improvements in predictive skill due to direct reconstruction of the global carbon cycle. Earth System Dynamics, 12(4):1139–1167, November 2021. doi:10/gnjh74.


Aaron Spring and Tatiana Ilyina. Predictability Horizons in the Global Carbon Cycle Inferred From a Perfect-Model Framework. Geophysical Research Letters, 47(9):e2019GL085311, 2020. doi:10/ggtbv2.


Roland Séférian, Sarah Berthet, and Matthieu Chevallier. Assessing the Decadal Predictability of Land and Ocean Carbon Uptake. Geophysical Research Letters, March 2018. doi:10/gdb424.


Zoltan Toth and Eugenia Kalnay. Ensemble Forecasting at NMC: The Generation of Perturbations. Bulletin of the American Meteorological Society, 74(12):2317–2330, December 1993. doi:10/dt6znw.


H. v Storch and Francis W. Zwiers. Statistical Analysis in Climate Research. Cambridge University Press, Cambridge ; New York, 1999. ISBN 978-0-521-45071-3.


F. Vitart, C. Ardilouze, A. Bonet, A. Brookshaw, M. Chen, C. Codorean, M. Déqué, L. Ferranti, E. Fucile, M. Fuentes, H. Hendon, J. Hodgson, H.-S. Kang, A. Kumar, H. Lin, G. Liu, X. Liu, P. Malguzzi, I. Mallas, M. Manoussakis, D. Mastrangelo, C. MacLachlan, P. McLean, A. Minami, R. Mladek, T. Nakazawa, S. Najm, Y. Nie, M. Rixen, A. W. Robertson, P. Ruti, C. Sun, Y. Takaya, M. Tolstykh, F. Venuti, D. Waliser, S. Woolnough, T. Wu, D.-J. Won, H. Xiao, R. Zaripov, and L. Zhang. The Subseasonal to Seasonal (S2S) Prediction Project Database. Bulletin of the American Meteorological Society, 98(1):163–173, January 2017. doi:10/cgwg.


Andreas P. Weigel, Mark A. Liniger, and Christof Appenzeller. The Discrete Brier and Ranked Probability Skill Scores. Monthly Weather Review, 135(1):118–124, January 2007. doi:10/b59qz5.


Matthew C. Wheeler and Harry H. Hendon. An All-Season Real-Time Multivariate MJO Index: Development of an Index for Monitoring and Prediction. Monthly Weather Review, 132(8):1917–1932, August 2004. doi:10/d489vd.


D. S. Wilks. “The Stippling Shows Statistically Significant Grid Points”: How Research Results are Routinely Overstated and Overinterpreted, and What to Do about It. Bulletin of the American Meteorological Society, 97(12):2263–2273, March 2016. doi:10/f9mvth.


Daniel S. Wilks. Statistical Methods in the Atmospheric Sciences. Volume 91 of International Geophysics Series. Academic Press, Amsterdam; Boston, 2nd ed edition, 2006. ISBN 978-0-12-751966-1.


S. G. Yeager, G. Danabasoglu, N. Rosenbloom, W. Strand, S. Bates, G. Meehl, A. Karspeck, K. Lindsay, M. C. Long, H. Teng, and N. S. Lovenduski. Predicting near-term changes in the Earth System: A large ensemble of initialized decadal prediction simulations using the Community Earth System Model. Bulletin of the American Meteorological Society, March 2018. doi:10/gddfcs.


Xiaojun Yuan, Dake Chen, Cuihua Li, Lei Wang, and Wanqiu Wang. Arctic Sea Ice Seasonal Prediction by a Linear Markov Model. Journal of Climate, 29(22):8151–8173, November 2016. doi:10/f88pgm.