Tuesday, July 18, 2006

The SENS debate, continued

As I promised to try to do, I read Estep et. al's submission to TR's challenge, the one the judging panel declared to be the "most eloquent", along with de Grey's response and the author's rebuttal. Estep et. al. misunderstand their role in the challenge from beginning to end. de Grey has been promoting SENS, a proposal for a new approach to combat aging, for a few years. From the beginning he admitted it was a controversial approach combining leading edge biology and an engineering mindset and asked experienced scientists to tell him if he was making any fundamental mistakes in the science. He has led and participated in symposia with experts and published papers in respected journals, after earning a Ph.D. in the relevant discipline. Technology Review solicited written submissions that would demonstrate that de Grey's program was "so wrong that it was unworthy of learned debate."

There are good reasons for practicing scientists to recognize the signs of pseudo-science, and it's appropriate for their first reaction to be outright dismissal when they recognize those signs. When a proposal garners enough attention that outright dismissal is no longer effective, it's still appropriate for most scientists to ignore it until there's more evidence in its favor. But at the point at which someone decides that a head-on confrontation is called for, the participants in the process should no longer expect to get away with derision; their role at that point is to review the science, show that the conclusions are at odds with established results, or demonstrate that they can't be disproved in principal, and thus are outside the bounds of science. Their goal should be to analyze the proposals, and summarize their evaluation in enough detail that other scientist can read their summary rather than doing the primary analysis themselves.

Estep et. al. spend an inordinate amount of time referring to studies of pseudo-science and its practitioners, and explaining in what ways de Grey and his proposal resembles them. But that's of no value at this stage in the process. Pontin (the editor of TR, and instigator of the challenge) and de Grey's other detractors have already pointed out that SENS isn't normal science; the challenge is to come up with an evaluation of the science, and Estep skimps on that.

Estep accuses de Grey of tangling his new proposals in with other scientists' results, making it hard to distinguish what is well-founded from what is not. But that's part of the normal process of science. Scientists build on others' work, expand it in some areas, and correct mistakes in others. In effect, Estep's charge is that scientists perpetrating fraud should clearly separate their new fraudulent work from others' well-founded work in order to make the investigator's job easier. This is nonsense. It is Estep's responsibility to examine the foundations of SENS and find parts that aren't justified or justifiable. If he can't, he has failed to undercut the science. That failure wouldn't by itself show that SENS is valid, but it would be a hurdle passed. Becoming accepted as established science is a process of continuing to get past the hurdles.

Estep shows no evidence that he understands the distinction that de Grey makes in referring to SENS as an engineering approach. de Grey is very clear in drawing the distinction:

Concerning the difference between scientists and engineers in mindset and motivation — as opposed to laboratory expertise — that I have often mentioned, Estep et al. expertly make my point for me by noting that the only reason they engineer model organisms is to find things out. To quote them: "If we could easily predict the outcome, why bother going through all the trouble of actually doing the engineering?". I wonder if Estep et al. think the Wright brothers built their airplane in order to discover whether it would fly? I personally suspect that they built it because they were confident that it would fly and they wanted to build something that would fly. Estep et al.'s oversight of this motivation is quite breathtaking to anyone who understands that, since aging causes immense suffering and death, it is something to be explored not for the sake of curiosity alone but with the goal of actually doing something about it.

In their response, Estep et. al. point out that the Wright brothers used the scientific process to test and evaluate the components they were building, and that they were systematic in their efforts to evaluate all the effects that mattered in getting into and remaining in the air. This response completely misses the point. Estep et. al. think that progress is made by systematically expanding the frontier of what is known. de Grey proposes to take what is known and build an effective mechanism to solve a problem, learning as he goes. The relevant question is whether we know enough at this point to start the process. The Wright brothers didn't know the answers when they started. They weren't satisfied with asking all the interesting questions, either; they asked the questions that were blocking their path to building a successful flying machine. Useful questions for critiquing an engineering proposal include: Do we know enough to get started? Is the cost estimate reasonable? Are there reasons to believe that there is no solution to the problem (within the budget)?

Estep et. al. focus on showing that some of the particular approaches that de Grey suggests aren't completely justified in the literature. But the heart of the SENS proposal is a search for solutions to 7 identified pathologies; de Grey has identified multiple approaches to each of them, and tried to give plausible detail about at least one solution for each problem. The useful counter-arguments are that these 7 aren't exhaustive, or that no (affordable) solution is available to one or more of the seven. Estep et. al. say that de Grey's list of 7 pathologies "unscientifically exclud[es] others"; the only exclusion I can identify in their paper is "unrepaired DNA damage in post-mitotic tissues", though they also mention "largely uncharacterized and undiscovered damage and pathologies". de Grey's original claim about the seven pathologies was that these seven had all been identified and characterized by 1982, and no new ones have become apparent since then despite all the advances and attention in this area, so "uncharacterized and undiscovered" looks like a clear miss to me. de Grey's response to the post-mitotic damage is to point out that he had addressed that issue in a draft paper that Estep has been asked to review. Estep talks about other items covered in that paper, but drops the thread on post-mitotic damage. I count that as no challenge to de Grey's claim that there are only seven pathologies to worry about.

de Grey wrote in his rebuttal:

Particularly incongruous is their accusation that I use the media to skirt expert criticism, when the SENS Challenge itself is my most conspicuous effort to do just the reverse, exposing the public reticence of SENS's off-the-record detractors and thereby forcing them to make their supposed case in print.

(de Grey has made other efforts to get gerontologist to put their objections on the record.) Estep responds thusly:

TR Editor Jason Pontin claims this challenge was initiated by him and all available evidence supports this claim. Furthermore, we have evaluated only a few of SENS' weaknesses, as did Warner and colleagues (Warner et al. 2005), and Aubrey de Grey's responses demonstrate beyond a shadow of a doubt that he is completely insincere in his wish for scientists to critique SENS.

Notice that Estep wrote in the original paper:

This SENS Challenge is itself part of the pseudoscience archetype, and it is simply a culmination of ongoing challenges made by Aubrey de Grey to opponents of SENS to prove him wrong. This is a classic attempt to subvert the scientific process, it is known to be typical of pseudoscience, and it is described as such in Dr. Friedlander's book, which predates SENS by several years.

So, Estep started this particular exchange by saying that the SENS Challenge is stereotypical behavior of pseudoscientists; de Grey rebuts that the challenge is an example of his having solicited criticism, at which point Estep backtracks and says that de Grey wasn't responsible for the challenge anyway. de Grey's Methuselah Foundation had matched Technology Review's prize for which Estep competed. Seems like de Grey is on solid ground in describing his role as helping to encourage debate.

Estep's rebuttal contains a number of explicit "predictions". That interests me because of the obvious application of prediction markets to monitor the argument. Most of the predictions are untestable or uninteresting: "de Grey's complaints that he isn't getting a fair hearing will continue", "Very soon it will be apparent to all [...] that SENS [...] constitutes overt scientific misconduct." Two exceptions are worth noting.

First, Estep et. al. say "[de Grey]'s only contribution to pharmacological inhibition and modification [...] of telomerase, therapeutic deletion of the entire telomerase gene [...] from the genome, will be recognized to be a crude biomedical fantasy. It will be abandoned by all sensible people--and even de Grey's co-authors will cease to [...] discuss it publicly." This is a very hard statement to write a clear prediction market claim for. We've tried before, on FX for claims that a consensus will arise, or that some research field will be dropped, but the claim details are hard to agree on and judge. I'll leave the prediction standing here as a reminder that de Grey's opponents can be judged to have made a valid prediction if, in your opinion, this subfield goes away later.

The second prediction is easier to adapt as a prediction market claim. The background is that de Grey is apparently accepted as an expert in the area of mitochondrial DNA (mDNA) mutation and repair. His proposal to combat progressive damage to mDNA is to ensure that Nuclear DNA (nDNA) can construct the few proteins (that's "allotropic expression") that the mitochondrion can't already get from the cell's nucleus. de Grey's response says

the shallowness of Estep et al.'s analysis is revealed especially starkly by their contradictory statements concerning allotropic expression in cell culture, which midway through their analysis they call an example of "assumptions and technologies that reside firmly in the realm of fantasy" but, in their summary, they call an example of "routine biology experiments."

Estep et. al. respond

successful allotropic expression of all 13 mitochondrial coding regions while maintaining mitochondrial and cellular function is a technology that resides in the realm of fantasy. Nevertheless, attempts to achieve these things are without a doubt routine biology experiments.

This makes it appear that Estep et. al. are willing to use the word "fantasy" to describe things that haven't been achieved yet. At least they didn't make any attempt to show that it couldn't be achieved.

Estep et. al. point out that de Grey bet someone publicly in 2000 that this would be known technology before the end of 2005, and it still hasn't been achieved. de Grey responds that he was counting on the imminent release of a paper showing the technique was possible (which was in review at that point) to spur further work. The paper only came out in 2005, so the follow-on work has only just started.

Estep's one explicit prediction is "We believe that this problem is [complicated] and it will remain unsolved for a very long time--if it is ever solved. Aubrey de Grey will continue to make excuses [...] to explain the lack of progress in this area." This, unfortunately, doesn't give a useful time scale, but de Grey's original projection can be extended to give a reasonable deadline.

Here is the background de Grey gives:

They lampoon my prediction from 2000 concerning AE, without mentioning that I made it assuming that Zullo et al.'s seminal breakthrough ( which I presented at the time I made the bet) would be published imminently in Science (where it was then in review), stimulating effort to perfect this approach; in fact, followup effort remained negligible until it was finally published in 2005. Thus, it is grossly misleading to suggest that my overoptimism arose from underestimating how hard AE is — and I fully explained this recently in a reply to Estep on a well-known mailing list (subscription required in order to review archives).

So in 2000, de Grey expected that imminent publication in Science would produce results by 2005. The paper was published in 2005, but not in a publication as prestigious as Science. Estep says it will be a very long time if ever. I'll propose a claim for the Foresight Exchange that it will be solved by 2012, which is longer than the 5 years de Grey originally expected, but gives some leeway for the reduced publicity the paper actually got. If it still seems to be 5 years away at that point, we'll have to conclude that de Grey's timeframes were way off.

Overall, I have to agree with TR's panel. Estep et. al. may have found some holes in de Grey's specific proposed therapies (it's hard for me to tell; unlike de Grey, Estep et. al. don't provide layman's versions of any of their technical arguments), but they didn't show that these holes make the entire effort unlikely to succeed, and they didn't show that de Grey's proposal doesn't address a useful goal.

As Marc Stiegler commented when I told him who was on the review committee, they seem predisposed to finding merit in de Grey's plan. Ventner is best known for his public commitment to develop technology to sequence the human genome before it was scientifically justifiable--and delivered. Rod Brooks' mantra at the Mobile Robot Lab was "fast, cheap, and out of control"; they were building stuff that worked rather than pursuing pure knowledge. Nathan Myhrvold is more focused on inventions than basic research, even though his academic credentials are impeccable.

added 12/15/2006

I created the mtDNA claim on FX in August.

4 comments:

Chris Hibbert said...

Well, it's not as if there's only one group talking, and all I have to do is examine their credentials to determine if they're smart enough that I should believe them. There are two groups, and they've both got credentials.

One is making an argument that a new approach to a problem is worth investigating. That side seems to have thought harder about how to solve the problem than the other side, which seems to be focused on basic research, and how things work in the body in the normal case. The second group has told the first group that it is ignoring the basic science, and the first group points to basic science that it thinks is in its favor. The second group made a lot of noise, and a respectable magazine challenged the critics to explain why the first group was wrong.

The submissions were judged not worthy. I wanted to know whether they had managed to undercut any of the basic arguments, since the judges didn't say what they believed and what they didn't. So I read the submissions.

I'm an educated lay person reading outside my field. So I can do a better job of evaluating what kinds of attacks were made than I can of deciding whose claims about the basic research are more correct. I wrote up what I learned by reading the articles. There may be real scientists who are convinced that SENS has absolutely no chance of working, but their best effort wasn't convincing to the judges or to me. And science isn't a matter of voting anyway.

I think de Grey is glad that the discussion has started. But the ball is back in the critic's court. I don't know whether the prize is still on offer if they can improve their argument, but they can still write papers explaining why it's not possible to clean up de Grey's 7 categories of damage or why doing so wouldn't help us to live longer. But I don't see how they can claim at this point that they've shown that SENS is worthless.

Anonymous said...

You say both sides have credentials. What are Aubrey de Grey's credentials, besides saying crazy things to get attention? Were you aware that what he calls his "Ph.D." from Cambridge isn't a real doctorate degree? What exactly is his training in biology or science? Who's lab did he do his work in?

I would think that an "educated lay person" who is thinking critically would be a bit more skeptical when someone (especially someone who looks like he escaped from the loony bin) claims people can live forever.

What I don't get is why these scientists even bother with de Grey.

Chris Hibbert said...

See http://www.sens.org/AdGbio.htm http://www.sens.org/AdG-CV.doc, and http://www.sens.org/AdGpubs.htm. He has a BA, MA, and PhD. I think the undergrad degree was in computer science, but he makes no secret of the fact that his background is engineering. He also makes no secret of the fact that he received his PhD by studying the subject on his own and convincing the faculty that he was qualified. He doesn't claim to have a laboratory background.

His list of refereed papers, and invited talks to scientific conferences is impressive.

People listen because other people who have heard him speak or read his work have recommended him. I was signed up for cryonics before I heard him talk, partly because I already found it plausible that the kind of work de Grey proposes will succeed.

Anonymous said...

Also perhaps you may be curious to see this new info on related topic:
New Books Discussing Aubrey de Grey Ideas
http://science-library.blogspot.com/2007/06/new-books-discussing-aubrey-de-grey.html
Shorter link:
http://tinyurl.com/29dt5o