Thursday, November 26, 2009

Eric S. Raymond on the East Anglia CRU's Global Warming Fraud

Eric S. Raymond ("esr") is an open source luminary partly because of his industry changing book The Cathedral & the Bazaar. As a fellow software engineer and open source advocate, I was curious what he had to say about the tree-ring circus that is Mann made global warming. He's written several posts on the topic which I excerpt below. esr frequently replies to his commenters, so I've included some of his more interesting comments. After reading Eric's reply to "krygny", I wondered if anyone's calling it the "hard-coded hockey stick".

On 11/21, esr, set the tenor for his upcoming posts in Hiding the Decline: Prologue:
For those of you who have been stigmatizing AGW skeptics as “deniers” and dismissing their charges that the whole enterprise is fraudulent? Hope you like the taste of crow, because I do believe there’s a buttload of it coming at you. Piping hot.
Comments on Hiding the Decline: Prologue:
Those who claim “scientific consensus” as a justification for any position are attempting to perpetrate a fraud, and have only themselves to blame when it blows back on them. The proper justification of any theory is not “consensus”, it is predictive power.

--

The most data could tell you is that average temperature is rising and CO2 is too. Well, except that average temperature isn’t rising – it was flat between 1998 and 2008, plunged sharply in 2008, and has not resumed the previous trendline. This is embarassing to AGW alarmists, since CO2 has kept rising and their theories require anthropogenic CO2 forcing to swamp anything that mere nature might be doing – and that’s manifestly not happening.

--
My point is that the data fails to meet the criteria the alarmists themselves have set. That is, they’ve been quite willing to interpret a short-period temperature rise between 1975 and 1998 as indication that we’re on a long-term trend with that slope, but when we get a decade of flatness after that they ignore it. It’s not responsive and not honest to point out that a decade is too short to mean anything unless you’re also willing to dismiss the previous 23 years.

Over longer timeframes, I don’t think there’s any statistically significant evidence that we’ve deviated off the very shallow warming trend following the last Ice Age. If you scrutinize the alleged data claiming otherwise, you keep finding noise and fraud.
On 11/23, he called for Open-Sourcing the Global Warming Debate, a call I readily agree with:
There is only one way to cut through all of the conflicting claims and agendas about the CRU’s research: open-source it all. Publish the primary data sets, publish the programs used to interpret them and create graphs like the well-known global-temperature “hockey stick”, publish everything. Let the code and the data speak for itself; let the facts trump speculation and interpretation.

We know, from experience with software, that secrecy is the enemy of quality — that software bugs, like cockroaches, shun light and flourish in darkness. So, too. with mistakes in the interpretation of scientific data; neither deliberate fraud nor inadvertent error can long survive the skeptical scrutiny of millions. The same remedy we have found in the open-source community applies – unsurprisingly, since we learned it from science in the first place. Abolish the secrecy, let in the sunlight.
Comments on Open-Sourcing the Global Warming Debate:
If it doesn’t happen, we’ll know they were scamming all along. Useful outcome either way.

--

And now, I think, those people are going to pay. Because it could still be that full disclosure will vindicate the AGW crowd, but having read the CRU material…I don’t think so. In all of the scenarios that are now plausible, those who bayed the loudest about “consensus” and howled for the persecution of “denialists” are now set up for a hard fall.

--
One of the reasons AGW flimflam angers me is that it crowds out sane, constructive environmentalism. An environmental lobby that really cared about saving the planet would be agitating for crash programs to replace the burning of fossil fuels with nuclear energy; buying up rainforest acreage to stem loss of biodiversity; funding research into better battery- and supercap-based storage technology so low-density renewable power sources could be aggregated into baseload power. But the envorinmentalists we have won’t do these things, because they’re fixated on the wrong problems and the wrong means of solving even those.

--

If the historical temperature data were generally known to be garbage (which I was pretty sure was true even before the leak), it couldn’t be used to justify public policy that is both bad and expensive – like the U.S.’s “cap-and-trade” bill in progress, which has so many giveaways and exemptions that it subverts its own ostensible purposes.
On 11/24, esr examined the code that, quite literally, creates the hockey stick graph in Hiding the Decline: Part 1 – The Adventure Begins:
This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1940s 1930s — see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.

All you apologists weakly protesting that this is research business as usual and there are plausible explanations for everything in the emails? Sackcloth and ashes time for you. This isn’t just a smoking gun, it’s a siege cannon with the barrel still hot.
Comments on Hiding the Decline: Part 1 – The Adventure Begins:
There was a brief note about it in a comment on someone else’s blog, enough to clue me that I should grep -r for ARTIFICAL. I dusted off my Fortran and read the file. Whoever wrote the note had caught the significance of the negative coefficients but, oddly, didn’t notice (or didn’t mention) the much more blatant J-shaping near the end of the series.

--

I have a closely related heuristic: any eco-related scare for which the prescription would result in a massive transfer of power to the political class is bogus.

--

krygny Says:
November 25th, 2009 at 8:02 am

Wait just a second. Explain this to me like I’m 12. They didn’t even bother to fudge the data? They hard-coded a hockey stick carrier right into the program?!!

ESR says: Yes. Yes, that’s exactly what they did.

--

...of course, they now claim that crucial primary datasets were “accidentally” deleted.

After reading some of the emails about evading FOIA2000 requests…accidentally, my ass.
On 11/25, esr asks: Will the AGW fraud discredit science?
Therefore…the next time we hear a ginned-up panic over some vast environmental crisis, the prudent thing to do will be to remember Mencken: “The whole aim of practical politics is to keep the populace alarmed (and hence clamorous to be led to safety) by menacing it with an endless series of hobgoblins, all of them imaginary.” It will be prudent to suspect that the science is probably already corrupted and demand extra-stringent scrutiny of it under that assumption.

...

And that brings us to process transparency. I discussed this with particular reference in Open-Sourcing the Global Warming Debate, but there’s another point that deserves attention. Strictly speaking, the rules of science require complete disclosure of all experimental methods, data, and analysis tools so that others can peer-review and replicate the work. We may find it an acceptable to relax those full-disclosure rules to some extent for corporations doing commercially-focused R&D. But that IPR exception should never be granted to scientists whose research touches public policy. Because the stakes are so much higher, disclosure standards must be as well.

If the “hockey team” had been required to make their primary datasets and modeling code available for unrestricted inspection, the AGW fraud could never have turned into a political monster. If Michael Bellesisles had been required to make all his primary data open for inspection, the fraud that was Arming America would never have won a Bancroft Prize. Sunlight is the best disinfectant, and full disclosure is the final and deadliest enemy of junk science.
Comments on Will the AGW fraud discredit science?
Matt Says:
November 26th, 2009 at 11:44 am

Corporate R&D is mostly engineering, rather than science. Thus, either it produces a working product (the precise definition of “working” may vary, but only the sponsoring organization’s definition and the market’s definition matter, and the latter only in the case of customer-facing projects) or it doesn’t…and if it doesn’t produce, then the methods behind it are irrelevant to anybody but the people who tried it.

The epistemological constraints on an engineer are thus easier to meet than those on a scientist. Peer review is still a good thing, but the integrity of the result isn’t dependent on it the way it is in the sciences, because the result is self-verifying.

ESR says. A telling point. Thank you.

--

>I am just wondering: Is AGW really junk science, because one team of experts (let’s say “experts”) did have an agenda?


No, it’s junk science because the agenda has driven systematic data suppression and fraud. It’s not the agenda itself that matters, it’s the breacjh in standards of scientific conduct. Those have been repeated and severe.
On 11/26, esr examines the Facts to fit the theory:
On 12 Oct 2009, climatologist and “hockey-team” member Kevin Trenberth wrote:
The fact is that we can’t account for the lack of warming at the moment and it is a travesty that we can’t. The CERES data published in the August BAMS 09 supplement on 2008 shows there should be even more warming: but the data are surely wrong.
Eyebrows have quite rightly been raised over this quote. It is indeed a travesty that AGW theory cannot account for the lack of warming, and bears out what I and other AGW critics have been saying for years about the fallaciousness and lack of predictive power of AGW models.

But the second sentence is actually far more damning. “The data is surely wrong.” This is how and where most scientific fraud begins.

Scientific fraudsters are not, in general, people pushing theories they know to be false. Outright charlatanism is not actually common, because it’s relatively easy to detect. Humans are evolved for a social competitive environernt and are rather good at spotting lies, except when they’re fooling themselves because they want to believe.
Keep an eye on Eric's Armed & Dangerous blog for more on East Anglia's Climate Research Unit (CRU) and the "science" of Mann made global warming.

Update: Instalanche! Thanks for the link, Glenn!! If you're here following climategate, checkout the links in the "Previously" section below. If you live in or around St Louis, MO, I hope to see you at the Tea Party at high noon on Kiener Plaza tomorrow (Saturday, 11/28).

Previously:

12 comments:

peter said...

I'm just guessing here, but if I were Phil the Hadley guy and I was "accidentally" deleting source data to stop others from uncovering my team's fudging.....well, I wouldn't actually delete it. I would copy it, hide the copy (at home, in the attic?), and delete the data from the server. I would want to keep a copy SOMEWHERE.

The threat of arrest and a search warrant may let us learn that these data have not disappeared from the face of the earth.

Like I say, just guessing.

The Whited Sepulchre said...

Here's what I'm having a good time thinking about:
Let's assume that East Anglia couldn't make the data fit, so they forced it. Guess what that means?
No one else can make the data fit, either.
It's not that hard.
If East Anglia U. (home of The Fightin' Alarmists) couldn't legitimately make their data fit the theory, then guess what?
Chances are, no one else can either. That includes NASA, Al Gore, and everyone else lined up for a payday.

rhhardin said...

Real science runs on curiosity.

That's a useful test to remember. It's often absent.

willem said...

This really isn't about AGW. This is about organized crime and racketeering. Why do I think we're looking at normed behavior deeply embedded in the fiduciary culture of the modern university?

Did we ever ask ourselves just who staffs, manages enables and champions ACORN and entities like it?

Perhaps we are looking at a septic culture in modern education centered in the universities themselves; a culture that has normalized systemic budgetary fraud and grant racketeering, data fixing, and other matters of social promotion, grade inflation and accreditation cohort fraud.

The prospective matter of AGW data and grant racketeering, and the fraud used to monetize it, may be the tip of the iceberg of failed academic rigor and an administration-enabled culture of academic fraud and data-for-hire schemes. I am coming to believe these modern university administrations are corrupt to the core. What are the public schools but an example of university work product.

Never has so much been paid by so many to so few for so little in return.

Time for RICO.

Jonathan Byron said...

If you want to believe that Earth has been cooling for the past decade, and that cooling is accompanied by accelerated melting of arctic sea ice, glaciers around the world, and a majority of ice shelves in Antarctica, go right ahead.

If you want to believe that humans can liberate carbon that was sequestered over millions of years into fossil fuels in a single century with no effects, go ahead.

Gregor Mendel's gathered data from his garden to establish his theory of inheritance. That data is suspicious - if it didn't involve outright fraud, there were massive errors of confirmation bias... the numbers are just too good to be true. But we don't have people shouting that the ideas of dominant and recessive traits are a hoax perpetrated by the crypto-communist illuminati.

The convergence of evidence supports the idea that human activity is changing the climate, even if a researcher or two engaged in unethical activity.

Mark30339 said...

The leaked data from CRU is at wikileaks.org

All the news has focused on the text in the emails about fudging data, evasion of FOIA, and corrupting the peer review process. The leaked data also has climate models. The revelation here is that the models have been found to be rigged to fraudulently create the hockey stick apocalypse.

Now with respect to Jon Byron's comment, contempt for Fr. Mendel and his 19th century data standards don't diminish the astounding prediction qualities of his theory.
In contrast, scientists of the 20th and 21st century have made a fine mess of things. France went into fission power big time, and happily produce energy with very little carbon release. The brilliant "consensus" of scientists that blocked the US from doing the same in the 70's and 80's fostered a fossil fuel carbon explosion. Now we should trust scientists again and pointlessly cripple our economy, while growth in China and India will reverse any carbon savings we create by 200% or more?

I say plant a lot more carbon scrubbing trees all around the world, and incarcerate scientists who fudge data or evade FOIA inquiries.

willem said...

Thank you for reinforcing my point, Jonathan.

Where wrongdoing has been normed its commission becomes trivial, admired by cohorts as evidence of sophistication and enlightened loyalty. IMHO, this goes fart to describe some of the hubris I see painfully exposed in the controversial emails.

I question the relevance of using Mendel as a rationalizing device. I doubt Fr. Mendel endeavored to destroy the careers of others, or engaged in willful deceptions or colluded with peers to enrich themselves at the expense of science, academic credence and human liberty.

How ironic Mendel's latter years were interrupted by European bureaucrats seeking more mechanisms of taxation, albeit from religious institutions. No telling how adversely his rigorous opposition to the tax vultures effected his body of work and controversies that followed.

Concretizing climate theory is as silly as being surprised to find the climate is changing. A good faith scientific inquiry requires something more than forming a secularized religious order of postmodern climatologists evangelizing for money and fame at the altar of the hardcode hockey stick.

willem said...

Ha ha. What a typo.

makomk said...

"On 11/24, esr examined the code that, quite literally, creates the hockey stick graph in Hiding the Decline: Part 1 – The Adventure Begins"

Except it doesn't. He's misrepresenting what that code does, and blocking any comments that point this out. What the code actually does - what all of the "hiding the decline" code does - is deal with a decline in temperature measurements derived from tree cores at very northern latitudes. This decline does not match up with any other temperature measurement in these locations, which is why it's such a headache.

The actual estimates of global warming don't use this data at all, with or without code to "hide the decline" - because there are better and more direct temperature measurements available.

All of this is already documented in scientific papers. It's not exactly a big secret. (Well, all the stuff that's actually used should be - there's likely to be code in there that was just written to test ideas out and that was never actually used for anything.)

humanbeingsfirst said...

Hello.

As a fellow engineer in a former life (bio on my website's About) I too have been curiously, but also forensically, trying to comprehend the multifaceted art of putting Science in the service of Empire.

In my analysis, I have discovered that it is more than some "bad-scientists" or "bad-science" conspiracy.

That art is Machiavelli.

Machiavelli means, as you obviously would well-know: "the mind of the state"!

That mind of state-craft well understands how human tendency for corruption and co-option can be put to good use as it is also well known that "deception is a state of mind".

The quote fragments on deception being the state of affairs for both the individual and the ruling establishment is by none other than the former Director of Counter-intelligence Operations for the CIA, James Jesus Angleton, 1954-1974, under whose watch were orchestrated all the "lone gunman" theories of officialdom for all the high-profile political assasinations that transpired in America, JFK, RFK, MLK, X, et. al., right under his watch.

The equivalent version being manufactured by the Mighty Wurlitzer's plants in the dissent-space for this monumental crime, to me at least,th appears to be lone "bad-scientists".

It is necessary to preempt that version becoming the officialdom's response when they do get around to unvieling it publicly.

Unfortunately, all those unraveling climategate minutae are contributing to it in various ways, including the primary researchers who have uncovered the mal-science behind what has known to be an agenda-setting-science for a very long time.

Therefore, I leave you a pointer to a comment I left to Prof. Judith Curry's editorialish essay, which in turn points to my paper which fleshes this out in fuller overarching context:

http://camirror.wordpress.com/2009/11/22/curry-on-the-credibility-of-climate-research/#comment-2236

If that comment disappears, my little analysis is also published here (alternate press of course):

http://atlanticfreepress.com/news/1-/12450-letter-to-a-co-conspiracy-theorist-reflections-on-modernity-climategate-pandemic-peer-review-and-science-in-the-service-of-empire.html



Thanks. And keep it up - dear fellow stay-at-home dad!
(see: http://prisonersofthecave.blogspot.com/2007/04/foreword-2005.html )



Zahir Ebrahim
Project Humanbeingsfirst.org

Gateway Defender said...

"The most data could tell you is that average temperature is rising and CO2 is too. Well, except that average temperature isn’t rising – it was flat between 1998 and 2008, plunged sharply in 2008..."

This concept is a bit of misdirection repeated ad nauseum by George Will and right-wing bloggers (e.g., Gateway Pundit).

Anyway, if the data are fudged why discuss it all?? Why would the data fudgers allow 1998 to be the warmest on record and lower in 2008? We can come back to that.

Check out: http://en.wikipedia.org/wiki/File:Satellite_Temperatures.png

The trend is for an increase of approximately 0.19 degree C/decade in average global temperature. Nothing in the temp record suggests otherwise.

Temperatures not only "plunged" in 2008 but also in the mid 80s, the early 90s, and in 1999 always to rebound. Annual global temperatures bounce up and down around an inexorably increasing mean.

For example in 1993 the temp anomaly (difference from long-term baseline) was + 0.06C and in 2003 +0.48C for an apparent increase of 0.42C! But based on the trend line the actual increase was more likely about 0.2C. The difference between 1997 (0.34C) and 2007 (0.52C) was +0.16C, between 1998 (0.52C) and 2008 (0.38C) was -0.14C, and between 1999 (0.25C) and 2009 (estimate ~0.5C) is +0.25C.

Selecting 1998 as the start date for comparisons is dishonest.

craig said...

"What the code actually does - what all of the "hiding the decline" code does - is deal with a decline in temperature measurements derived from tree cores at very northern latitudes. This decline does not match up with any other temperature measurement in these locations, which is why it's such a headache."

Properly speaking, it is not a "headache" to be managed by massaging the data. What it is, is a prima facie indication that tree core measurements are inherently unreliable indicators of temperature. In other words, when tree cores fail to correlate to actual temperature data from 1960 onward, the logical inference is that they should correlate poorly to actual temperatures prior to 1960 as well.