Reading the "ClimateGate" Emails
On November 19, 2009, a 61 megabyte file called "FOI2009.zip" started to circulate on the internet. This contained a directory of over 1000 emails sent to and from people at the Climate Research Unit, University of East Anglia, UK, plus supporting documents and software. Check a BitTorrent site for "FOI2009.zip" if you want a copy, or you can find a searchable index of the email at http://www.climate-gate.org/.
CRU is one of the major climate research centers, and the people sending the messages are the biggest names in climate science. The revealed messages have been fairly embarrassing to the people involved. Phil Jones, the director of the CRU, has temporarily stepped down while it's being investigated.
Right-wing global warming "deniers" are having a field day with it. Mainstream press and the lefty blogs are taking a "move along, nothing to see here" attitude. Many commenters don't seem to have read the emails themselves. They are just repeating the same fragments over and over in the usual echo chamber.
I've been interested in climate issues for awhile now, and have written a couple of long pieces on where I think things stand. If you had to describe my position in a single sentence, it would be "the current data, models and theories of climate aren't solid enough to say one way or the other if humans are warming the planet." You can find my previous items here:
What's in the Mail
The earliest email is March 7, 1996, and the latest, November 12, 2009. Most of them are to, from or CC'd to Phil Jones, Director of CRU, but many different authors are represented, and some of the emails do not even CC Mr. Jones. Also, despite the juicy comments that have been reprinted, the vast majority of the mail is day to day details on preparing research papers or chapters for the IPCC documents. It's dry as dust. See Email 1139932579.
An interesting question is where this all came from. It's usually described as the result of hacking the Climate Research Unit (CRU). I doubt this is right. These are a very carefully selected set of emails, with none of the personal messages or administrative junk you'd normally find in an archive. It also just doesn't have the "nyah, nyah" shock value and sloppiness of most hacker efforts. I think this is more likely to be an inside job.
Despite the name, and some speculation along those lines, I don't think this set of messages was selected as part of an official Freedom Of Information Act reply. No administrator, lawyer or CRU member would include emails that insult other members of the community, especially when there's no actual climate science mentioned. On top of that, the emails themselves show that CRU had no intention of answering the FOI requests.
Since there's no single person listed as sender, recipient or CC on all of the messages, it's hard to see who could have collected all of these. Phil Jones himself comes closest, so perhaps this is extracted from his personal archive. It's hard to believe he'd have kept some of these though. It will be interesting to see what develops.
Climatology Is Hard
If you want to go straight to the accusations against the CRU and my take on them, go here.
But first, I think we need some background, including a discussion of the data that climate science relies on. In my opinion, the "smoking gun" in the messages are not the badmouthing of critics, the attempts to manipulate peer review, or the refusal of the FOI requests. Instead, I was surprised by their own doubts about the quality of climate data.
If we see warming today, and want to blame it on human activity, it would help if we could answer two questions.
To answer either of those questions, we need historical climate data. Knowing current temperatures and following the recent trend isn't enough.
Data is Scarce
The first problem is that the historical data is scarce, and the farther back in time you go, the worse the problem gets. The map below compares the warming of the 1930s with the warming of the 1990s. The gray parts of the map are places where there is no temperature data in the 1930s, and so no comparison can be made. You can see that much of the world is a bit warmer now than in the 1930s, but some of it is colder. The temperature in the U.S. is about the same now as then.
Data is missing in South America, Africa, the Middle East, Central Asia, the Arctic and Antarctic. The Year 1900 would look even worse, almost completely blank outside North America and Western Europe.
I haven't plotted the sea data from CRU because it's heavily interpolated to show the entire world. This is a problem, since there's no way they have data for the entire world. Instead, they have temperatures collected by working sailors traveling shipping lanes (no global warming research vessels or automated buoys in 1930!). The modern shipping lanes look like this:
Anything far off these lanes won't be in the data. Again, there would be little at the poles, and not many points from the middle of the Pacific Ocean. The Southern Hemisphere would be poorly covered.
What's the point of these maps? Any graph that shows global warming this century has to have used this data. All of the climate research centers have access to the same land weather stations. Their climate graphs and models only differ in how they process the data.
The climate researchers can't create data where there is none. They can't create a true average temperature for 1930 with a map this skimpy. A big part of their trends could very well be determined by how they compute these global temperatures. If the true data for 1930 showed a hot spot over Africa or Antarctica, it would completely revise the graphs of 20th century temperatures. All of those graphs should be displayed with wide error bars, with the size of the error increasing as you go back in history.
In addition, any claim that this decade is the "warmest in a century", or "warmest in 1000 years" is stretching things. We don't have enough data to compare even to 1930, when we know it was comparably warm in the U.S..
There are several types of data used in climate research: satellite data, land temperature stations, sea surface temperatures, and proxies. Let's look at each, and see what the CRU emails have to say about them.
Satellite data is widely regarded as the best data there is, because it covers the entire planet, and seems independent of data gathering issues or "urban heat island" effects. I thought this was the gold standard, but there's an email that makes me doubt it:
Even though much of the differences [between satellite and ground temps] may now be apparently explained, it's still a terribly messy job. The satellite system wasn't designed to measure tropospheric temperatures, the calibration and orbital decay and retrieval algorithm and all the other technical issues are ugly, and nobody knows how much the lower stratospheric cooling ought to have infected the upper troposphere, among other points one might make.
Good or not, the real problem with satellite data is that it only goes back to the 1980s. This means we don't have any satellite coverage of prior warm periods like the 1930s to use as comparison. Instead, we compare standardized ground temperatures with standardized satellite temperatures. Since the data was collected with completely different types of instrument, the comparison will only be right if your standardization techniques are right.
Land Temperature Stations
These are the ground stations at airports and cities, used for weather reports and forecasting before satellites. Some go back to 1880 in the U.S. The rest of the world is very different. The farther you go back, the fewer there are. For earlier periods in the 20th century, the coverage is minimal outside North American and Western Europe.
There are issues with this data as well. Stations have been moved, the time of day readings were taken sometimes changed, and there are breaks in the data. If a city grows up around the station, the "Urban Heat Island" effect may bias the readings upwards over time. A website, http://www.surfacestations.org/ claims that two thirds of the sites they have checked do not meet the standard for accurate readings. How much to correct for this is an open question. How much to correct for Urban Heat Island effects is an open question as well.
At 17:01 on 01/06/2009, Phil Jones wrote to Chris Folland,
Sea Surface Temperatures
Temperature has been measured by ships at sea for decades. Here's one of the emails on Sea Surface Temperature (SST) data. This is a writeup for publication (not sure by whom), forwarded from Gavin Schmidt to Phil Jones:
... Like almost all historical climate data, ship-board sea surface temperatures (SST) were not collected with long term climate trends in mind. Thus practices varied enormously among ships and fleets and over time. In the 19th Century, simple wooden buckets would be thrown over the side to collect the water (a non-trivial exercise when a ship is moving, as many novice ocean-going researchers will painfully recall). Later on, special canvas buckets were used, and after WWII, insulated 'buckets' became more standard - though these aren't really buckets in the colloquial sense of the word as the photo shows (pay attention to this because it comes up later).
Maritime records back to 1900 or before (handwritten, probably) would have been digitized. Then each record would have been coded for the type of bucket used. Then each measured temperature would have been "standardized" to adjust the temperature record based on type of bucket, to make them all uniform.
This all matters because if you adjust a bucket temperature by 1 degree, you are adjusting it by more than the warming signal you are looking for. If you get that wrong (systematically wrong, because of your software or assumptions) you can bias the data upwards. There's even an email about this:
On Oct 16, 2008, at 6:52 AM, Phil Jones wrote:
Proxies, including Tree Rings
Prior to 1880, you have few temperature records from thermometers. So they try to reconstruct climate from "proxies" -- natural events, like the growth of trees (shown in the width of the rings), that are related to temperature.
The ideal case would be something like a California Redwood tree, sitting in a forest undisturbed by people or nearby cities, for 2000 years. If the growth of the tree were affected strongly by temperature, and that location had a strong global warming signal, you'd be set. Unfortunately, that's not the case. Here are some issues with tree rings:
At 11:57 11/16/2006, Keith Briffa wrote:And this email from a critic, quoted in Email 1256760240:
At 10:34 10/02/2009, Donald Keiller wrote to Kieth Briffa:
The Unfortunate Timing of Climate Records
Here's the "hockey stick" graph from the IPCC report, based on tree-ring data from Michael Mann:
Although this graph is in dispute, it still illustrates a fundamental problem. Proxy records like tree rings are used up to around 1850-1880. There's debate on which to use, and whether global temperatures show the "Medieval Warm Period" and the "Little Ice Age". But no one thinks man-made global warming is happening during this period.
Right around 1900 we see a sharp warming in this graph and others. Unfortunately, that's right when the "instrumental record" (land and sea temperatures measured with thermometers) starts. Wouldn't it be nice to have a proxy like tree rings that exactly matched the modern record from thermometers? But apparently, we don't have that. The kinds of long-lived trees that have been cored for records don't grow near populated areas where there are long temperature records. Cores are coming from the high Rockies or Siberia, and are very difficult to calibrate to modern records.
Finally, we see a second sharp warming in the 1980s. But this is when the instrumental record is replaced with the satellite record! Again, it would be really nice if satellite data went back far enough to see the cold period from 1950-1980, but it doesn't. And in fact, the instrumental records starts to thin out again after 1980, as weather forecasting switches to satellites from ground stations. There are fewer ground stations now than there were in 1975, and big parts of Africa have dropped off the land-based maps again.
What you would want ideally is a single record that covers the last 1000 years, showing all the historical events and extending into the modern period where it could be compared to thermometers and satellites. We don't have that. Instead we have a patchwork of records from different sources, that all have to be standardized with lots of fiddly little adjustments. This is just asking for statistical errors and biases to be added into the data, producing a trend where there is none. The Accusations Other blogs have covered the juicy bits of the emails, so I'm not going to. You can find lots of details out on the net, or look at the emails yourself. Let me give a summary of what CRU has been accused of, and my take from reading the email.
CRU is Evading Freedom Of Information Act Requests
It's a matter of record that Steve McIntyre and others made FOI requests for the data held by CRU. These requests were refused. The emails make it clear that CRU had no intention of answering the requests. They found excuses to refuse the requests and that was the end of it as far as they were concerned. There's also a single email from Phil Jones to Michael Mann asking him to delete emails concerning the preparation of the IPCC Fourth Assessment Report (AR4).
If they get into any legal trouble, it will be because of this issue. The more interesting question is why they refused. Possibilities include:
There are two other reasons supported by the emails.
At 01:17 03/12/2008, Ben Santer wrote to Tom Wigley:Note though that although Steven McIntyre is an outsider and not a climatologist, the other people named (Christy, Douglass and Singer) are authors of peer-reviewed papers published in major climatology journals.
At 16:50 02/10/2009, Malcolm Hughes wrote to Keith Briffa:
It's not clear to me from reading the emails when the tree cores were done. Some of the messages make it sound like it was done in the 80s and 90s, which would be a long time to hold on to data. The email above makes it sound more recent.
Of course, this would all be acceptable if climate science were still just an academic exercise. It's not so defensible when trillions of dollars and a whole new energy infrastructure are on the line. You see that with a lot of the emails. They want to do science the traditional way, and aren't prepared to deal with investigations, FOI requests, making data and software public, and answering blogs. They don't seem to understand this strange urgency their critics have...
Also, CRU could have (and should have) just released what they had -- raw sea temperature records, the software used to adjust it, and their software for interpolating the data to cover the entire world. I think they didn't because it was not in very presentable form. To meet any kind of "publication" standard, lots of documentation would have to have been written, software cleaned up (and "tricks" documented), etc. It would have been a major piece of work.
Apparently, CRU is promising to do this now, with the results available in a few months. I think to answer their critics, they should dump all the rough-and-ready versions they have now out to the net. While they work on it, the wider community will be happy to sift through it all.
CRU is Corrupting the Peer Review Process
The CRU group has been accused of throwing their weight around to get papers by their critics rejected by scientific journals. Then they sneer at skeptics saying they have no published peer-reviewed papers.
There's some evidence in the emails that they tried, and some of the blogs make it sound like they succeeded to some extent. It's hard to decide exactly what they were thinking though. There are a range of explanations.
Reading the email, I think the biggest problem is that they have no "loyal opposition" internally, or even in the field -- people who would take warming seriously, but force them to validate their results more carefully. None of the CRU emails that I've read come across as "Devil's Advocate" sort of messages. It's mostly true believers there, and they are a bit defensive.
Trying to get reviewers fired, etc., is not anyone's idea of how science is done, but it's not clear to me that it has made any difference. Papers refused in prestige journals can still be submitted elsewhere. The problem is that no one seems to take any challenges to the basic idea of warming seriously at this point.
CRU is Unable to Reproduce its Own Results
In addition to the email, there is a "documents" directory, and it contains a file named "HARRY_READ_ME.txt". Speculation is that this is by Ian "Harry" Harris, a CRU member. It documents an attempt to get some of the CRU software and datasets in usable condition, after the original authors have left, or perhaps just after years of neglect. The comment "Argh!" can be read at many, many points.
You can go to open-source guru Eric S. Raymond's blog for scathing analysis of this code, ClimateGate issues, and global warming in general. Many sarcastic comments by programmers.
I'm not going into this in detail because this item is too long already. Here's what I take away from the discussion:
I'm not trying to give them a pass here. There's really no excuse for not documenting every step of the process. It's not good science, and it's certainly not good public policy to base a huge decision on science that can't even be documented properly, or reproduced by other teams.
CRU is Altering the Raw Data
A lot has been made of CRU "hiding the raw data" or "altering the raw data", or throwing it away after they've produced their "value-added" data. The fact is, there is no single source of pure "raw" data, and the raw data isn't usable anyway.
Datasets come in from many different sources and CRU "standardizes" them to put them all on the same temperature scale, so they can all be compared to one another, or fed into computer models that generate climate predictions. It's this "standardized" data that CRU keeps, and the original, unstandardized records that it has thrown away.
These same issues pervade all of the data. Satellite data is adjusted. Land and sea measurements are adjusted. Tree-rings and other "proxies" that go far back in time are very adjusted. They should have kept original sources (sea temperature logs, for example) and a description of what was done to each reading to standardize it. For tree rings, even the selection of which trees to core affects the "raw data". For ground temperatures, the siting of each thermometer (whether it's in a city, or next to an air-conditioning unit) is "raw data." It should all be online, with each number linked to software and a description and images. Dream on!
The original data is probably still out there somewhere (and it's not clear how much CRU has actually thrown away.) A team that wanted to replicate everything CRU has done would digitize it, code it for source, adjust it (with their own software) and produce a new temperature record. That would be a lot of work.
So did they really "alter" the data in the way that skeptics are claiming? I have two issues from reading the emails.
First, the use of tree-ring data seems very problematic. As I discussed in the Climatology section above, there are a list of questions about whether tree rings can be used to say anything about global climate. And as far as I can tell, trying to use this data is the source of all the "artificial" corrections in their software.
Second, there's this email that seems odd:
At 23:25 on 09/27/2009, Tom Wigley wrote to Phil Jones:
It's hard to say what they are getting at here. Since he mentions the 1910-1040 warming (which is nearly as large as the 1980-2000 one), I doubt this is what he means by "1940s blip." Still, it does sound like they are trying to impose an artificial correction to the sea temperature data so that a smooth warming curve, without any inconvenient "blip" is shown on the result.
This is the opposite of starting with the data and looking for trends. This sounds like imposing a trend (or smoothing it, at least) on existing data. But without context, it's hard to be sure.
The Bottom Line
I have a lot of reservations about climate science and the quality of the data. The emails confirm some of those reservations. But does the ClimateGate archive really prove anything about the science?
The archive (probably leaked) is damaging to the people involved. It makes them look nasty, petty and defensive. It raises many suspicions about whether, as Global Warming True Believers, they have maybe found what they are looking for.
For example, when the tree ring data comes out "wrong", you could just keep processing it until it looks "right" -- that is, agrees with all the other data you have. Or you could take the portions of the record you think bolster your case, and ignore the modern parts that don't. If you Truly Believe, and have put a lot of work into gathering tree ring data, you are less likely to say "tree rings are worthless as records of climate" or "we just don't understand why these numbers diverge from other records."
It's not clear to me that CRU has really distorted the process much, or made a mess of climate research. I think the FOI and peer review issues are both overblown. They are very bad form, and perhaps legal trouble for some of the people involved, but they don't say much about the science of Global Warming.
Going forwards, I don't think this release will help the science much. Activists on both sides will dig in their heels and refuse to admit problems. No climate scientist will write an email saying what he honestly thinks of a paper or a person for awhile. In fact, I expect a lot of email has been deleted around the world this week!
Perhaps there will be more openness. I wouldn't be surprised if some scientific journals bend over backwards to publish some skeptical papers. But the bigger problem is that the science is "settled" in the minds of so many researchers. That is premature, in my opinion.
Let me finish with an analogy. You've probably seen these Discovery Channel documentaries on dinosaurs. In one, a guy with a silly hat and wild hair will get on and declare that T-Rex was a predator, no doubt about it! And he'll point to bones and teeth, and then say something like "anyone who says otherwise just isn't doing good science!"
Then the next guy with wild hair will come on and say that T-Rex was clearly a scavenger, and point to the absence of any fossil bite marks or trackways. He'll argue that T-Rex would have fallen over trying to chase something. And just look at the silly little arms! "Anyone who thinks T-Rex was a predator has watched too many movies."
This is all very amusing (I keep expecting it to turn into a Monty Python sketch), and matters to almost no one outside the perhaps 100 people in the world who have ever measured a dinosaur bone.
Now imagine that activists descended on these people, saying they were going to spend a few trillion dollars cloning a T-Rex, and absolutely had to know, right now if T-Rex was a predator or scavenger! Most of the scientists would freeze like deer in headlights. Some of them would mumble about uncertainties and lack of data, but the activists and politicians don't want to hear it.
Instead, the few True Believers would step forward and say "T-Rex has been proven by science to be a predator (or a scavenger.) I'll stake my scientific reputation on it!" Policy makers would pick the side that was politically popular. They would be influenced by the public, who think "of course, it's a predator -- I saw Jurassic Park!", and whatever industry concerns there were.
After a winner was picked by politicians, funding would dry up for the opposition. Without grants, they can't do their own fieldwork to get raw data. Without data, they can't write papers. Without publications, they can't get grad students. Without money or students, they can't get more data. The science would be declared "settled" and skeptical work would be shut down.
That, unfortunately, is what I think has happened to climate science. These groups
are still acting like academics -- sniping at rivals, keeping minimal records
and writing amateur software as if this were a purely academic issue. But the politics have
taken over and the stakes are too high to continue business as usual.
For more, see Free The Memes!
Have a Comment?