Yellow Flags On A New Methane Study

A forthcoming paper in the influential Journal of Geophysical Research has measured concentrations of methane and other alkanes in air near oil and gas operations in Weld County, Colorado, and has used that to infer rates of methane leakage from natural gas production in the region. Their results, which were reported last week by Nature and the Associated Press, point to much higher methane emissions than consensus inventories have previously found. Indeed some have read the results as confirming the highly controversial estimates of methane leakage published last year by Cornell professor Robert Howarth and his colleagues.

The new study is far more sophisticated and careful than what’s been published so far. It deserves serious attention. But after several days of working through the data and analysis, I’m skeptical of some of the results and interpretations. That doesn’t mean that I think that the paper is wrong – it’s premature to say that – but, given the attention it’s been getting, I think it’s worth putting some of the potential problems out in public.

One note before I dive in: Some of my observations may seem a bit oblique. Recent experience has started to convince me that, when there’s a problem with peer-reviewed scientific work, it’s important to correct the formal scientific record, rather than just responding on the web. Alas, many scientific journals have bars on publishing material that’s already been made public elsewhere. In this case, I may end up submitting a technical “Comment” to JGR, so I’m going to have to be careful not to preempt myself here. I apologize to readers for whom that makes this post difficult to evaluate.

Now on to the four issues that I want to flag. The first two have to do with the paper itself; the second two are problems with how it’s being interpreted.

Assumptions about the composition of leaked gas.

The authors need to make some assumptions about the ratio of methane to propane in vented gas in order to do their analysis. They do that in three different ways in order to bracket uncertainty. There’s no reason, however, to assume that their three cases bracket the range of reasonable possibilities.

Alas, it turns out that their results are highly sensitive to the choice of ratio. If, for example, the methane-to-propane ratio in vented gas is a lot higher that in any of their three candidates, the implied rate of methane leakage is much lower than what they estimate. If, on the other hand, the methane-to-propane ratio in vented gas is lower than in any of their three candidates, the implied rate of methane leakage is even higher than their alarming estimates. This, to me, is the biggest problem in the paper, and introduces a lot of uncertainty around the results.

Are we really talking about gas wells?

The paper presents its final results as if it’s found out that there’s a lot of leakage from natural gas production. But a large fraction of the wells it uses to come up with chemical profiles for vented gas aren’t gas wells. In particular, the sorts of methane-to-propane ratios that it uses aren’t actually representative of gas wells – they’re representative of condensate (basically oil) wells with significant amounts of associated gas. In won’t get into the weeds here, but when I rework the numbers, I come to one of two conclusions: there’s a lot of methane leakage in the area they’ve studied, and it’s coming from condensate wells, or there isn’t much methane leakage there, but it’s coming from gas wells.

Why isn’t this just nitpicking? Because people are already taking the paper and trying to extrapolate it to genuine gas wells in places like Pennsylvania. It’s far from clear, though, that one can do that, if we aren’t actually talking about gas wells in the first place.

A peculiar spin on uncertainty.

The paper presents several different estimates of methane leakage (based on different methodologies) all with clearly reported uncertainties. Reporting has emphasized the full range of possibilities. But the range over which the different estimates overlap is actually pretty small, and it is close to the lowest, and least alarming, possible methane leak rates. It’s not clear to me why the upshot of this paper (setting aside the other issues I’ve flagged) isn’t that leakage rates are probably in this lowish area.

To understand why I’m puzzled, let me sketch an analogous example. Through decades of work, we know that the mass of the neutron is about 1.67E-27 kg. Now let’s say that I come up with a new experiment that shows that the mass of the neutron is between 1.5E-27 kg and 3.34E-27 kg. No one should write an article declaring “Mass of Neutron May Be Twice Previously Agreed Level”. They should look at my work and say: “Given that other methods peg the mass of the neutron at 1.67E-27 kg, and this new experiment is consistent with that, we should probably stick to what we know”. They should chalk the upper end of my experiment’s range up to the fact that my experimental method happens to be imprecise.

The same is true for this paper. Just because one approach in the paper can’t chop uncertainty down doesn’t mean that methane emissions might be stratospheric – it means that you should juxtapose that approach with other ones in order to narrow down the range of possibilities.

Apples, oranges, and the Howarth study.

The authors of the JGR paper report a “best guess” estimate that 4% of Weld County gas production is leaked. Howath tells Nature, “I’m not looking for vindication here, but [the JGR] numbers are coming in very close to ours, maybe a little higher”. Indeed the similarily between the JGR and Howarth numbers is one of the main reasons that the JGR paper has attracted attention.

But this comparison doesn’t work. In Howarth’s percentage calculation, the denominator is expected gas production from wells drilled during the period under study; in the JGR paper, that denominator is gas production during the period under study itself. These are different numbers, which means we’re talking about different ratios.

Bottom line? There’s a lot of fantastic observational data in the JGR paper. I suspect that there’s more that can be done with it to shed light on methane leakage. For now, though, I’m not ready to rely on its results.