2.1 Dataset

We analysed the historical runs of 50 atmosphere–ocean GCMs (AOGCMs) and the Representative Concentration Pathway (RCP) 8.5 scenario runs of 42 AOGCMs participating in CMIP5 (Taylor et al., 2012). A single ensemble member, r1i1p1, was selected for each model, except for CESM1-WACCM (r2i1p1), CSIRO-Mk3L-1-2 (r1i2p1) and EC-EARTH (r8i1p1). This is because the member, r1i1p1, of CESM1-WACCM and CSIRO-Mk3L-1-2 was not available and the temperature change from r1i1p1 of EC-EARTH was over 2 standard deviations of the averaged changes from the 42 models in more than 60 % of our target regions. In the following, the full set of the multi-GCM ensemble indicates the 50 historical runs when we assessed the ability to reproduce the historical climate (CMIP_{Full_Hist}), while the full set indicates the 42 future projections which are estimated from both historical and rcp85 runs when we discussed the future projections (CMIP_{Full_Future}).

We compared the simulations of the subsets of GCMs used in ISIMIP and CORDEX with the full ensemble. ISIMIP used four GCMs for their various impact assessments: GFDL-ESM2M, HadGEM2-ES, IPSL-CM5A-LR and MIROC5 (Frieler et al., 2017). On the other hand, CORDEX used the subset in which the combination of GCMs were altered for each defined region. The number of GCMs used in each of the defined regions is listed in Table 1, and each GCM is listed in Table S1 in the Supplement. The regional classification used to investigate the regional performance and the projection was based on the classification in CORDEX shown in Fig. S1 in the Supplement. In this study, we focused on global land area, considering the importance for both programs because of the relevance to human activities.

Table 1Number of CMIP5 models used in the CORDEX regions.

Download Print Version | Download XLSX

The analysis periods were the years 1986–2005 (but 1985–2004 for HadGEM2-CC and HadGEM2-ES) for the historical runs and the years 2081–2100 (but 2080–2099 for MRI-AGCM60 and CESM1-WACCM) for the RCP8.5 runs. Monthly mean temperature and precipitation data over these periods were interpolated onto a $\mathrm{2.5}\times \mathrm{2.5}{}^{\circ }$ grid for each model. The grid cells with the temporal mean precipitation of <0.1 mm d⁻¹ were defined as “too dry” grid cells and the precipitation values for these cells were not considered. This is because we expressed the precipitation change in a ratio and thus the ratio tends to be large at too dry cells even when the change is quantitatively extreme small. It is difficult to explain the meanings of such a large ratio physically. By applying the threshold, grid cells indicating an extremely large ratio, for instance, 100 %, were excluded. The total number of the excluded grid cells is approximately 5 % of all target cells as an average over the used members.

To validate the model representations, we compared the simulated estimates with the observed datasets. With respect to precipitation, Sun et al. (2018) highlighted differences among the observational datasets. Consequently, to avoid a misreading of the model performance due to such discrepancies, we used seven different precipitation products that covered the global land area over the period of interest. The observation products were the Climatic Research Unit Timeseries (CRU) v.4.01 (Harris et al., 2014) for temperature and precipitation, and the following for precipitation only: the global unified gauge-based analysis by NOAA Climate Prediction Center (CPC) v.1.0 (Xie et al., 2010), the Global Precipitation Climatology Centre (GPCC) full data reanalysis v.7.0 (Schneider et al., 2016), NOAA's Precipitation reconstruction over Land (PRECL) v.1.0 (Chen et al., 2002), the CPC Merged Analysis of Precipitation (CMAP; Xie and Arkin, 1997), the Global Precipitation Climatology Project (GPCP) v.2.2 (Huffman et al., 2015) and the Multi-Source Weighted-Ensemble Precipitation (MSWEP) v2.1 (Beck et al., 2019). The difference among the observations was calculated as the deviation from GPCC as the reference. To quantify the ability to reproduce spatial patterns of the observations, we used the skill score proposed by Taylor (2001; hereafter referred to as skill score) as follows:

where R is the spatial correlation coefficient between reference observation and simulation, σ is the standard deviation of simulation normalized by the reference spatial pattern and R₀ is the maximum correlation attainable. The value of R₀ was assumed to be 1 here. In addition to the skill score, we also evaluated the magnitude of the model bias. Using both metrics enables the assessment of both the spatial pattern and the bias magnitude.

2.2 Coverage of uncertainty and random selection

Coverage was estimated from a comparison between the full uncertainty range of the projections made by two model sets, which was defined by McSweeney et al. (2015) as a fractional range coverage, FRC. In this study, we computed the regionally averaged projections for each model, and then the FRCs were estimated using the regional averages. The FRC from the regional averages (FRA) was defined as the fraction of the maximum–minimum range of the uncertainty in the regional averaged projections from a subset of CMIP_{Full_Future} (R_Sub) to the range from CMIP_{Full_Future} (R_Full), as follows:

The range of R_Sub was computed from the ISIMIP and CORDEX subsets and also from arbitrary subset samples we generated. From the comparison with the arbitrary samples, we can investigate how well the ISIMIP and CORDEX subsets captured the uncertainty range of projections. With reference to MJ2016, our arbitrary samples were generated by randomly selected n models without repetition from CMIP_{Full_Future} 10 000 times, where n is the sample size of subsets in ISIMIP (n=4) or CORDEX (n depends on the regions; see Table 1). Then, the variance of the FRA was estimated from the 10 000 random subset samples of CMIP_{Full_Future} and compared with the FRA from the ISIMIP and CORDEX subsets.

3.1 Performance in reproducing the historical climate

Using model biases and skill scores, we evaluated the historical climate reproduced by the GCM subsets used in ISIMIP and CORDEX. The GCM subsets used in ISIMIP and CORDEX are hereafter referred to as the ISIMIP subsets and CORDEX subsets respectively. For the evaluations, we also used two high-performance subsets: one is composed of models with lower bias than the 50th percentile (median) of the CMIP_{Full_Hist} biases; the other is composed of models with a higher skill score than the median of the CMIP_{Full_Hist} scores (referred to as CMIP_lowB and CMIP_highS respectively). The models included in the high-performance subset are shown in Fig. S2. B(v(E)) and S(v(E)) indicate the regional mean biases and skill scores for variable v and ensemble subset E respectively.

Figure 1 shows the model bias associated with the annual mean precipitation in the 14 CORDEX regions over a 20-year period. Compared with the maximum values of B(P(CMIP_{Full_Hist})) for the precipitation (v=P), the maximum values of B(P(ISIMIP)) and B(P(CORDEX)) are clearly small, especially in the Mediterranean (MED), Southeast Asia (SEA), and the polar regions. The spreads of B(P(ISIMIP)) and B(P(CORDEX)) in MED are within the spread of the discrepancy among the observations, which suggests that the model selection works effectively to select models with high ability to reproduce the observed regional mean precipitation quantitatively. However, compared with the high-performance subsets, some models in the ISIMIP and CORDEX subsets have a bias exceeding the maximum values of B(P(CMIP_lowB)), or B(P(CMIP_highS)) in some regions, despite the small number of models used in ISIMIP and CORDEX. Therefore, our results indicate that less-biased models could be selected than those currently being used. The difference in the spread between the ISIMIP and CORDEX subsets has different characteristics region to region, and this is partially related to the overlapping of model members used across ISIMIP and CORDEX. For example, in five regions of Central and South America, Europe, Africa and South Asia, the CORDEX subsets include more than three of four ISIMIP models and the ensemble is large (Table S1). As a result, the variance of biases estimated from the CORDEX subset covers that from the ISIMIP subset. Especially in Europe, the difference of the variance between the CORDEX and ISIMIP subsets is large, and it is found that the models used in the CORDEX subset but not included in the ISIMIP subset make the variance increase. Focusing on the regions where the CORDEX subsets include only two models in the ISIMIP subset, the variance from the CORDEX subset tends to be larger than that from the ISIMIP subset, especially in the regions with large ensemble of the CORDEX subsets, like North America, SEA and Australasia. By contrast, the variance from the CORDEX subsets is relatively small in the regions with a small ensemble of the CORDEX subsets, like MENA and Central Asia. In East Asia, the variance is small in CORDEX despite using seven models in contrast to four models in ISIMIP, indicating that the biases from the seven models are almost the same.

Figure 1Normalized annual mean model precipitation bias over land from the GPCC reference data (in percent). The bias was normalized by the regional average of GPCC data. The whiskers of the box plots show the range between the maximum and the minimum biases. The boxes and the lines within the boxes indicate the 25th to 75th percentile range and the median respectively. Green plots indicate the deviations of six observation data from the reference data: CRU, GPCC, PRECL, CMAP, GPCP and MSWEP. The other plots indicate the model bias in the full set of 50 CMIP5 model set (black), the model sets with a smaller bias than the median of biases of the full set (yellow), the model sets with higher Taylor's skill score than the median of the scores of the full set (orange) and the model sets selected for ISIMIP (blue) and CORDEX (red).

With respect to the spatial pattern of the annual mean precipitation, ISIMIP and CORDEX incorporate some models with a worse score than the minimum value of S(P(CMIP_highS)) (Fig. S3). That is to say, ISIMIP and CORDEX subsets include members showing a spatial pattern of low similarity to that of observations. S(P(ISIMIP)) and S(P(CORDEX)) fall within the observational spread only in the Arctic.

We also assessed model performance for the annual mean temperature (v=T). The maximum values of B(T(ISIMIP)) and B(T(CORDEX)) are smaller or equal to the maximum value of B(T(CMIP_highS)) (except for the CORDEX subsets in East Asia and North America), but are larger than the maximum value of B(T(CMIP_lowB)) (Fig. S4). The spread of B(T(ISIMIP)) is covered by that of B(T(CORDEX)) in the same four regions as the bias in the precipitation except for Europe because of the overlapping of model members used. The spreads of B(T(ISIMIP)) and B(T(CORDEX)), however, resemble each other compared with the precipitation bias in most regions, indicating that CORDEX used models with a quantitatively similar performance to ISIMIP, despite using more models than ISIMIP except for Central Asia. Both subsets included models with a worse score than the minimum value of S(T(CMIP_highS)) in 85 % of the regions (Fig. S5). Therefore, relative to CMIP_highS, the subsets can quantitatively represent the observed temperature as a regional average well, but the spatial pattern represented by some members in the subsets does not resemble that of the observations.

Even though the model selections conducted in ISIMIP and CORDEX narrow the spreads of model bias and the score from CMIP_{Full_Hist}, the largest bias and the worst score from the ISIMIP and CORDEX subsets are distributed beyond the biases and the scores from high-performance models in the full set.

3.2 Uncertainty range of the projected changes in annual mean temperature and precipitation

Future projections obtained from the ISIMIP and CORDEX subsets were compared with those from the full set, and also from high-performance models, as with the evaluations in Sect. 3.1. Because the small bias or high skill score models used in this section are composed of the models included in CMIP_{Full_Future}, we refer to these as CMIP${}_{\mathrm{lowB}}^{\prime }$ and CMIP${}_{\mathrm{highS}}^{\prime }$ instead of CMIP_lowB and CMIP_highS. Projected changes in annual mean temperature and precipitation are designated by ΔT(E) and ΔP(E) respectively.

Figure 2 shows the uncertainty range of the projected temperature increments, calculated from the average over the 20-year period for each model. Although ISIMIP used fewer models than CORDEX, the uncertainty range of ΔT(ISIMIP) is greater than or equal to that of ΔT(CORDEX) except for Central and South America, South Asia and Australasia. The uncertainty ranges of ΔT(CMIP${}_{\mathrm{lowB}}^{\prime })$ and ΔT(CMIP${}_{\mathrm{highS}}^{\prime })$ broadly cover the range of ΔT(CMIP_{Full_Future}), suggesting that the bias and skill score are not good emergent constraints for reducing the uncertainty of ΔT in this study, though previous studies have showed the reduction of their projection uncertainties (e.g. Smith and Chandler, 2010; Bracegirdle and Stephenson, 2013; Bracegirdle et al., 2013; Simpson et al., 2016). This is because the spatial pattern for the temperature is quite similar among the models, and then the model selection using the score hardly has an impact on the reduction of uncertainty. On the other hand, the difference in the bias between the full set and the subset is large. The previous studies have suggested that the performance of the present climate simulations is not necessarily related to the uncertainties of future projections (e.g. Knutti, 2010; Shiogama et al., 2011). We expected such a relation between the quantitative performance and the future change in this study.

Figure 2Annual mean temperature increments in the future climate projection (K). The whiskers of the box plots show the range between the maximum and the minimum increments. The boxes and the lines within the boxes show the 25th to 75th percentile range and the median respectively. Box plots indicate the increment in the full set of 42 CMIP5 models (black), the model sets with the top 50 % of the CMIP5 models for the bias (yellow) or Taylor's skill score (orange) and the model sets selected for ISIMIP (blue) and CORDEX (red). The top 50 % of the CMIP5 models cannot be plotted over Antarctica because of the lack of CRU reference data.

The uncertainty range associated with the projected change in annual precipitation is shown in Fig. 3. Compared with ΔT in Fig. 2, model selection has a large impact on the reduction of the uncertainty in ΔP, as was also found by MJ2016 using five GCMs used in the fast track of ISIMIP. The subsets of ΔP(CMIP${}_{\mathrm{lowB}}^{\prime })$ and ΔP(CMIP${}_{\mathrm{highS}}^{\prime })$ cover 70 % and 60 % of the full range of uncertainty from CMIP_{Full_Future} as the average over 14 regions respectively, and cover the full range in Australasia (yellow and orange plots in Fig. 3). The largest difference between the coverages from ΔP(CMIP${}_{\mathrm{lowB}}^{\prime })$ and ΔP(CMIP${}_{\mathrm{highS}}^{\prime })$ appears in East Asia. Therefore, we need to note that, when a model's performance is the condition used to select subsets, the uncertainty changes depending on which evaluation index is used, for example whether we use the bias or the skill score. The CORDEX subsets capture more than 50 % of the full range in eight regions (Europe, MED, Africa, SEA, Australasia, Central America, South America and Antarctica). On the other hand, the ISIMIP subsets capture less than 60 % of the full range in all regions. In 11 regions, the CORDEX subsets capture a wider range than the ISIMIP subsets, a result markedly different than for ΔT, where both CORDEX and ISIMIP have relatively large coverage as seen in Fig. 2. Therefore, the subset of four models used in ISIMIP2b has difficulty capturing the uncertainties in regional precipitation change. This result is the same as stated using the subset of five models used in the fast track of ISIMIP discussed by MJ2016, despite two of the five models being changed.

Figure 3As for Fig. 2, but for the projected change in annual mean precipitation scaled to the regional mean temperature increment over land (% K⁻¹).

The uncertainty range (maximum–minimum) is narrowed by using the subsets, but the interquartile range of ΔP(CORDEX), IQR(ΔP(CORDEX)), shows a high coincidence with IQR(ΔP(CMIP_{Full_Future})), as well as with IQR(ΔP(CMIP${}_{\mathrm{lowB}}^{\prime }\left)\right)$ and IQR(ΔP(CMIP${}_{\mathrm{highS}}^{\prime }\left)\right)$. The maximum–minimum range of ΔP(ISIMIP) also captures IQR(ΔP(CMIP_{Full_Future})). Therefore, the CORDEX and ISIMIP subsets can capture the average tendency of the change projected by the 25th to 75th percentile of CMIP_{Full_Future}. In addition, the median of the uncertainty range is similar between the CORDEX subset and CMIP_{Full_Future}. In Central Asia, the full range of ΔP(CORDEX) remains below the 25th percentile of ΔP(CMIP_{Full_Future}), while the maximum–minimum range of ΔP(ISIMIP) adequately covers the IQR(ΔP(CMIP_{Full_Future})). Thus the three models of the CORDEX subset in Central Asia underestimate the average tendency of the change projected by CMIP_{Full_Future}, despite that, differing from ISIMIP, CORDEX can select suitable models for examination of climate change in Central Asia.

3.3 Comparison of uncertainty of the projected changes using randomly sampled models

We investigated whether the ISIMIP or CORDEX subsets were more suitable for capturing the uncertainty range obtained from CMIP_{Full_Future} by comparing the fractional coverage of uncertainty, FRA, of each subset with those of 10 000 randomly sampled subsets of CMIP_{Full_Future}. As a result, the ISIMIP subset (four models) shows high coverage for the temperature change in all regions compared with the random samples and low coverage for the precipitation change in more than 60 % of all regions. By contrast, the CORDEX subset yields relatively wide coverage for the temperature and precipitation changes, but this depends on the number of models used.

Figure 4 illustrates FRA of the ISIMIP and CORDEX subsets (referred to as FRA_ISIMIP and FRA_CORDEX respectively) in each region. Along the x axis, the name of regions is arranged in ascending order of the number of models used in CORDEX. The number of models used in CORDEX is indicated in each set of parentheses after the name, and by contrast, the number in ISIMIP is four in all regions. The y axis indicates FRA of the uncertainty from each subset relative to that from the full set. The bar represents the distribution of the FRA values obtained from the possible 10 000 random samples (FRA_Random). The blue bar represents the distribution using the subsets with four models as large as the ISIMIP subset (FRA_{Random_I}), and the red bar represents that with the same number of models used in CORDEX (FRA_{Random_C}). The ends of the bar indicate the lowest and highest values of FRA, and the ends of the bar with a dark colour and horizontal line in the bar denote the 25th and 75th percentiles and the median respectively.

Figure 4Coverage performance of the ISIMIP and CORDEX subsets compared with the range of the full set of CMIP5 models for (a) annual mean temperature increment and (b) precipitation change scaled to the regional mean temperature increment. Blue and red dots indicate the coverage (FRA) in ISIMIP and CORDEX respectively for each region. Blue bars indicate the spread of FRA when four models, as in ISIMIP, are selected randomly in 10 000 times. Red bars indicate the spread when randomly selecting the same number of models as in CORDEX, e.g. 10 models in Central America. The full range of the coloured bars indicates the minimum-to-maximum coverage of the FRA spread. Dark blue and red bars indicate the 25th to 75th percentile range. Horizontal lines in the dark blue and red parts indicate the median. Numbers in parentheses are the number of models used in CORDEX.

Download

For the temperature change, ΔT, FRA_ISIMIP and FRA_CORDEX (blue and red dots respectively) exceed 60 % in 13 and 10 regions respectively (Fig. 4a). However, FRA_CORDEX is located around the 25th percentile or less of FRA_{Random_C} (the bottom of dark red bar) in MED, East Asia, SEA, Europe and the polar regions where FRA_CORDEX is lower than FRA_ISIMIP. In the region with larger model ensemble in CORDEX, FRA_CORDEX tends to be less than the median of FRA_{Random_C} (horizontal red line). On the other hand, FRA_ISIMIP is typically around the 75th percentile (the top of dark blue bar) or higher than the median (horizontal blue line) of FRA_{Random_I} for all regions.

A relatively high coverage, above ∼50 %, is shown on FRA_CORDEX for both changes of temperature and precipitation in eight regions when using nine models or more, except for temperature in Antarctica (Fig. 4a, b); that is to say, the CORDEX subset captures more than half of the range from CMIP_{Full_Future}. The value of FRA_CORDEX for ΔP is lower than that for ΔT. A high coverage of more than 70 %, however, can be gained by the CORDEX subset for ΔP in MED, South America, Europe, Australasia and Africa, which also indicates a high coverage compared with the median of FRA_{Random_C} (except for Europe; Fig. 4b). In half of the regions, FRA_CORDEX are in the range of the 25th percentile or less of FRA_{Random_C} (the four regions of Asia, MENA, the Arctic and North America). In Central and East Asia and North America, FRA_CORDEX is smaller than FRA_ISIMIP, even though CORDEX has the advantage of selecting suitable models for the region and also more models can be used, especially in East Asia and North America. The ISIMIP subsets in Antarctica and Australasia show a larger coverage than the 75th percentile of FRA_{Random_I}, but the FRA_ISIMIP of 60 % is less than that for ΔT. In more than 60 % of all regions, FRA_ISIMIP is less than the median of FRA_{Random_I}; the averaged FRA_ISIMIP over all regions is 33 %.

From the FRA distributions estimated from the possible random samples regarding both changes, ΔT and ΔP, the IQR of FRA_{Random_C} itself rises toward a FRA of 100 % as larger model ensembles are used. When random samples are composed of a subset with 15 models, as large as subsets in CORDEX-Africa and CORDEX-South Asia, the 75th percentile of FRA_{Random_C} is more than 90 % in ΔT (Fig. 4a). In addition, the width of the IQR for ΔT is narrowed with increasing the number of models. The relationship between the number of models and FRA is clearly evident in ΔT because there is a small difference in R_Full among regions for ΔT compared with ΔP (Fig. 2), and thus the larger model ensemble results in an increase in FRA_CORDEX and FRA_{Random_C}. And also, we found that the probability of selecting model subsets with a low coverage was higher for precipitation than for temperature, even if the number of models selected increases.

From Fig. 4, the subsets with nine models or more can capture the uncertainty of projections in both temperature and precipitation widely, implying that there is a heterogeneity in the dataset in a different number of models (Gutowski et al., 2016). We explored whether a similar tendency can be obtained in the other regions when the number of models changed. The same approach was performed by MJ2016. They estimated the coverage of the uncertainty in each of the grid cells for each number of models to investigate the change of the widest coverage performance in the global or regional average depending on the number of models. On the other hand, in this study, to consider making better use of the current subsets, we investigated how the coverage changes with changing the number of models from the current model members.

Figure 5 shows the change of coverage performance with the number of models changing in each region. When the number of models is larger than the current number, we added models randomly selected to the current members. By contrast, when the number of models is less, we removed models randomly selected from the current members. Here we focused on the median of the FRA values obtained from the possible 10 000 random samples, meaning the FRA value obtained with a possibility of 50 % when selecting subsets randomly. For the temperature change, the median exceeds 60 % in all regions when changing the number of models from the current four ISIMIP members to seven members (Fig. 5a). The median above 60 % is also obtained in 13 regions (except for Antarctica) when changing the number from the current CORDEX members to nine members. For the precipitation change, the coverage in nine members is above 50 % in 10 regions and in 12 regions by changing the number of models from the current members in ISIMIP and CORDEX respectively (Fig. 5b). Even when using nine members, the median is less than 50 % in the four regions of MENA, Africa and South and East Asia for the change of number from the ISIMIP subset and in the two regions of MENA and North America for that from the CORDEX subset.

Figure 5Change of coverage performance of the ISIMIP and CORDEX subsets depending on the number of selected models in each region for (a) annual mean temperature increment and (b) precipitation change scaled to the regional mean temperature increment. As Fig. 4, but the x axis denotes the number of selected models.

Download

The IQR for ΔT shifts to a high FRA smoothly with the number of models in all regions. By contrast, the IQR for ΔP sometimes gets large suddenly and/or shifts sharply, for instance, MENA and Africa. The discontinuous change is caused by a large variance of ΔP from each model member. That is to say, when there are model members indicating a large change ratio relative to the other members, the coverage largely differs depending on the inclusion of the member with the large ratio or not. The change amounts, ΔT, are similar among the model members and the variance is small. Thus, the FRA increases with the number of models and the IQR also increases smoothly. To prevent selecting the subset with a large change of coverage dependant on a model with an extremely large or small change amount, investigating the variance of the projections in each region is needed when the number of models is decided.