The Intergovernmental Panel on Climate Change's 2007 report from the group working on global warming's impacts contained at least one error. "Glaciers in the Himalaya are receding faster than in any other part of the world (see Table 10.9) and, if the present rate continues, the likelihood of them disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate," the report notes.

A detailed analysis by the InterAcademy Council (IAC)—a composite board of many of the world's national scientific bodies, such as the National Academy of Sciences in the U.S.—of the history behind this statement finds that at least three reviewers, including the government of India, challenged it as the 2007 report was being drafted. And it was undercut in the report's very next sentence: "Its total area will likely shrink from the present 500,000 to 100,000 km2 [square kilometers] by the year 2035 (WWF, 2005)."

"100,000? You just said it will disappear," ecologist David Saltz of the Jacob Blaustein Institute for Desert Research Institute at Ben-Gurion University of the Negev noted during the IPCC's review process, according to records kept by the IPCC that were among the data reviewed by the IAC.

Nevertheless, both sentences made it into the final report—an example of potential problems in the IPCC process that the IAC was charged in March to review.

The Himalayan glaciers mistake "came from just not paying close enough attention to what review editors commented," economist Harold Shapiro of Princeton University noted, calling for review editors to identify the most important points that come up and focus lead authors on addressing those criticisms. "It may have been they were overwhelmed by the sheer number of comments. That's just my inference." Shapiro chaired the IAC effort, the results of which were released August 30 at a news conference at the United Nations in New York City. The review drew from data ranging from IPCC participants' testimony to an online survey open to the public.

The IAC concluded that "the overall structure of the IPCC assessment process appears to be sound, although significant improvements are both possible and necessary," the report's authors wrote.

Calling for "fundamental reforms" in the IPCC process, the IAC noted that the undertaking is becoming enormous—in addition to wrangling 194 government representatives and more than 3,000 scientists, the IPCC authors had to deal with some 90,000 comments on their most recent report. So, for example, the IAC report recommends that IPCC lead authors not be required to respond to comments that are "purely editorial," in Shapiro's words.

Other reforms include: forcing governments to comment before plenary sessions (government changes consistently slow the release of IPCC summaries); a clear and codified conflict of interest policy; and a permanent management structure consisting of a "small" permanent executive committee that includes members from "outside IPCC or even outside the climate science community" as well as a full-time chief executive in the form of an executive director who is a senior scientist.

"We need a management structure at the top," IPCC chair Rajendra Pachauri agreed at a press conference at the U.N. following the IAC's report release. "It's still a very informal structure. I'm very happy that the IAC has recommended that this structure, in somewhat different shape and form, be formalized."

Fundamental flaws?
The Himalayan glaciers error was drawn from non-peer-reviewed findings, known as "gray literature," which nonetheless has a place in the IPCC reports, the IAC found. "We found that such material, which can include technical reports, conference proceedings, observational data or model results, often is relevant and appropriate for inclusion in the assessment reports," said Shapiro at the press conference. The key to their fair use is making guidelines for such use both clearer and more strictly enforced "to ensure that unpublished and non-peer-reviewed literature is adequately evaluated and appropriately flagged in the reports."

The IAC report is at least the sixth review of IPCC processes, procedures and outcomes in the past year. None, including the IAC, have found cause to question the underlying conclusions based on the best available science on climate change from the most recent assessment. "Man-made drivers are responsible for most of the climate change that we have experienced in the past 50 years," explained climatologist Thomas Stocker of the University of Bern, co-chair of the working group on the physical science of climate change for the next IPCC assessment, at the press conference following the IAC's report release. "This result has never been challenged."

The IAC also calls for leaders of the IPCC, including the chairs of the various working groups, to serve for only one assessment, although this did not reflect any judgment on current leadership, according to Shapiro. Pachauri is now overseeing his second such assessment.

How to handle uncertainty
Most importantly, perhaps, the IAC concludes that the IPCC must be strict in its handling of uncertainty in climate science. Shapiro and his colleagues recommend that the IPCC adopt a "level of understanding" assessment of the likelihood of any given outcome or scientific statement as well as a clear description of the basis for that assessment. Such a model is employed by the IPCC working group on mitigating climate change and delineates whether there is high or low agreement among experts on the finding as well as a large or small amount of evidence for that finding. Such an assessment could then be backed by a more quantitative assessment of uncertainty when appropriate, such as in the case of directly observable measurements like CO2 concentrations in the atmosphere.

"I have been involved with a group that is reworking the uncertainty guidance for [the next assessment report], and we are trying to fix those issues," says economist Gary Yohe of Wesleyan University, who was interviewed by the IAC for its report and served as a lead author for the IPCC. "It is difficult, and one size does not fit all…but we think we will have something better to offer authors soon."

The IAC-recommended approach would have prevented some of the other errors that found their way into the overall report. "The authors reported high confidence in statements for which there is little evidence, such as the widely quoted statement that agricultural yields in Africa might decline by 50 percent," the report's authors note, one of three explicit errors that have been identified in the more than 4,000 pages of the combined IPCC reports, including one caused by an inaccurate submission of data by the government of the Netherlands. The latest peer-reviewed science suggests the IPCC also erred at least two more times in underestimating the speed and scale of melting glaciers and sea ice.

Once such reforms are in place, such as a clearly defined conflict of interest policy, the report's authors also suggest further opening the IPCC to other viewpoints—ranging from providing travel grants for developing world experts to participate in the IPCC as well as opening the doors to private companies. "Their research and support of the process could significantly expand the available knowledge base concerning adaptation and mitigation options," the report notes.

That is a conclusion backed by the current IPCC chair Pachauri, who recently was criticized for potential conflicts of interest deriving from consulting work undertaken for private companies, although a formal apology was published by the U.K.'s Sunday Telegraph, which first revealed such criticisms in an article it later retracted. "Why should we not provide business advice?" Pachauri asked at the press conference. "They have to be part of the change if there is a change. I functioned purely in an advisory capacity."

The IPCC report was "a success and served society well"
The report's authors also went beyond their specific review of the IPCC processes and procedures to suggest further reforms that might strengthen future outcomes, including setting up a Web-based system for including and synthesizing the latest research or further staggering the release of the various working group's reports to allow those on impacts and mitigation to make full use of the report from the working group on the physical science of climate change.

"Errors did dent the credibility of the process. Trust is something you have to earn every year," Shapiro said. "We think what we recommend will help restore some of this trust."

But, overall, the "IPCC's assessment process has been a success and served society well," Shapiro noted. "The assessments have put IPCC on the world stage, raised public awareness of climate change, and driven policymakers to consider options for responding to climate change." That's a conclusion backed by entities ranging from the Dutch and U.S. environmental agencies (pdf) to independent academic researchers, who have all completed reviews of the IPCC's scientific claims in the past year.

The increasing scrutiny—and expanding workload—does not seem to have diminished scientists' and other experts' interest in participating in the next IPCC report, due in 2013. "The number of nominations to work on [the fifth IPCC assessment] increased almost 50 percent to 3,000," Pachauri noted. "From those we selected 831 experts compared with 559 in 2004 for [the fourth IPCC assessment]. What a tremendous show of support."

And, in the end the IPCC's errors, in this case, were errors of degree rather than fundamentals, although the IPCC retains the goal of "eliminating every questionable statement or error that it's humanly possible to prevent," said biologist Chris Field of the Carnegie Institution for Science's Department of Global Ecology at Stanford University and co-chair of the working group on impacts for the next IPCC assessment. Himalayan glaciers may not disappear by 2035 but the U.S. Geological Survey 2010 report on glaciers throughout Asia found that most of them are retreating as a result of climate change.

"The IPCC sits at the intersection of science and policy and, in many ways, it represents a significant social innovation," Shapiro said. "Most of our key recommendations are aimed at helping IPCC manage this increasingly complex process and doing so under the gaze of a public microscope."
 
As for what happens now that the report is public, Pachauri said that "it is for the IPCC and all the governments to decide when they want to implement the recommendations and which ones they want to implement." The IPCC will next meet from October 11 to October 14 in Busan, South Korea.