Science and Steamrollers: How Research Stories Can Go Off the Rails [UPDATE]

In a blog post earlier today I did a bit of a dissection, from my personal perspective of a big story that came out of AACR this week (see earlier post here).


No sooner had I hit the "upload" button, I became aware of a wonderfully written piece that was hitting the 'blogosphere' at the same time, from one of the authors already quoted in my earlier piece - Erika Check Hayden.

From the Blog "Last Word on Nothing" she wrote a piece entitled "What the ‘limits of DNA’ story reveals about the challenges of science journalism in the ‘big data’ age.


As I did in my earlier post, I will enumerate her key messages:

1. Science consists of more and more “big data” studies whose findings depend on statistical methods that few of us reporters can understand on our own..
2. Challenges in the news business are ratcheting up pressure on all of us. 
3. We are only as good as our sources. 
4. It’s becoming more difficult to trust traditional scientific authorities. 
5. Beware the deceptively simple story line.
6. Getting the story right matters more than ever. 

As much as I am tempted to do so, good practice prohibits me from just cutting and pasting her whole blog post here, but I do encourage you to go back and read what she says on each of these. Please do; it is worth the click and the read.

As you can see, a number of her points very much parallel my own, but rather than feel smug about that, it inspires me to keep doing this blog. And I aspire to writing on a subject as cogently and eloquently as she does.

Science and Steamrollers: How Research Stories Can Go Off the Rails

Earlier this week I wrote a post about “personal genomics”. Actually I wrote 2 posts the same day (part 1 and 2) but who’s counting <smile>?

Short Recap:

In the second article (read it again, here) I talked about a new study that was announced at the Annual Meeting of the American Association for Cancer Research (AACR) in Chicago (meeting website here), and published simultaneously in the journal Science Translational Medicine.  This study, authored by a team at Johns Hopkins in Baltimore and led by world-famous researcher Dr. Bert Vogelstein, was entitled The Predictive Capacity of Personal Genome Sequencing. The researchers used disease registries from several countries, and looked at data from over 50,000 identical twins and tracked how often one twin or the other developed one of 24 different diseases.

Because twins have identical genomes, by comparing the results of one twin to the other, the question was asked: how much do their genes predict any increased chances of getting a disease. It was concluded that most of the twins were at average risk for most of the 24 diseases, pretty much the same as anyone from the general population. 

In other words, the authors suggested that widespread use of genome sequencing will likely provide very little useful information to enable  prediction of future disease.

Because of the fanfare around this paper, and not in the least because of the reputation of the research team in general and Dr. Vogelstein in particular, this study became instantly of interest in the mainstream  media, and in the social media universe, especially on Twitter.

One of the first to pick up on this story was Gina Kolata, writing for the New York Times. Ms Kolata is, to my mind, a seasoned, very well respected and well-known science journalist and her article in the Times (Study Says DNA’s Power to Predict Illness Is Limited) offered both a recap and this caveat:

“While sequencing the entire DNA of individuals is proving fantastically useful in understanding diseases and finding new treatments, it is not a method that will, for the most part, predict a person’s medical future.”

Another well-known journalist, Robert Bazell posted an article on MSNBC entitled "Gene tests: Your DNA blueprint may disappoint, scientists say" that carried pretty much the same cautionary message: 

“If everyone has a complete gene profile, a small number can learn they have a great risk for something.  But for most, the information is minimally significant.”

I confess that, given the reputations of Dr. Vogelstein and of the mainstream journalists covering this story, that I too felt a bit deflated in that moment, and said in my own blog post: 

“Bottom line, it seems to me, is that we really have to be more careful than ever about exposing ourselves to privacy, confidentiality, insurability and other legal and ethical dilemmas, especially if the risks might outweigh the gains in many, if not most, cases.”

What Happened Next?

At least I was open-minded (or prescient?) enough  to have ended my post with the caveat that:

 “Clearly there is no definitive pronouncement to be made one way or the other yet - it is far too early days for that. But it is good to have these debates with our eyes wide open.”

Indeed, as is so often the case, closer inspection with eyes wide open and sober second thought reveals that there is more to this story than meets the eye.

Actually, perhaps it might be better said that there may be “less” to this story than meets the eye...

Initially, I was rather surprised to see a very vigorous, but negative reaction from a number of other  journalists and scientists alike, especially in the ‘Twitterverse’ not only to the Vogelstein study, but to the media attention that it was getting. Some of the critiques explored how the study was flawed, or at least how its conclusions might have been flawed given the design of the study.

But the main critique that I read loud and clear from several independent sources was essentially that this result was to be 100% anticipated, and that geneticists and other molecular biologists  have been saying this for some time. In other words that ‘there is no news here’.  And so they were perplexed at why the study had been positioned to have been some brand new discovery. Worse, they were very concerned that the rush to judgment of the mainstream media, by lacking critical perspective, might seriously set back genomic research progress by unfairly damning this whole area of research without benefit of having asked many critical (and contrary) questions. 

The Other Side of the Story

I’m sure there must have been many more, but I will highlight 3 excellent pieces that have appeared since the original story broke and the original media attention flourished.

One of the very best was written by Erika Check Hayden in a blog post entitled “DNA has limits, but so does study questioning its value, geneticists say” published in Nature’s Newsblog. In that post she writes that:

 “Geneticists don’t dispute the idea that genes aren’t the only factor that determines whether we get sick; many  of them agree with that point. The problem, geneticists say, is not that the study ... arrived at a false conclusion, but that it arrived at an old, familiar one via questionable methods and is now being portrayed by the media as a new discovery that undermines the value of genetics.”

She went on to list 5 main critiques which I will enumerate here, but which you should go back to the original article to read the details:

1. This study critiques the power of genomic medicine but does not contain any genome data. 
2. This study is beating a dead horse.
3. The mathematical model used in the study is unrealistic.
4. The study doesn’t correct for errors that can affect twin studies.
5. The media coverage of the study could weaken support for genetic research.

 To me, another  very well written and compelling “rebuttal” was penned by Luke Jostins on the Blog “Genomes Unzipped” in an article entitled “Identical twins usually do not die from the same thing”. 

In his post he ponders why “a not particularly original or particularly well done attempt to answer a question that many other people have answered before, got so much press (including a feature in the [New York Times]).”

He goes on to try to answer his own question, and the insight is commendable:

But of course, the reason is relatively obvious. All of the papers I linked to there are by statistical geneticists ... and never came with a press release or an attempt to talk to the public about them. The message, to those who can read them, is clear and well established – genetic risk prediction (or any form of risk prediction) will never be able to perfectly predict disease incidence, and will never replace diagnostic tests. But the fact that the results of Bert Vogelstein’s study seems to have come as a surprise to people, when it comes as no surprise to us, shows us that we have failed in one of our primary duties to keep the public informed about the results of our research. The paper’s failure as a work of statistical genetics stands in contrast to its success as a work of public outreach. If we are annoyed that a bad paper got the message across, then we should be annoyed with ourselves that we never communicated our own results properly”

And finally, a blog post yesterday from Paul Raeburn in the Knight Science Journalism Tracker entitled “What everyone should know about genome scans”, not only provides a very nice summary of the debate, but goes one step further to pose questions about the role of the press and of journalists who cover science and research that gets exceedingly complex. In some very insightful comments, Mr. Raeburn asks, for example:

“The question here is how reporters might have suspected these criticisms and produced better stories–or how their reporting might have done a better job of uncovering the potential pitfalls of the study. Few reporters are qualified to assess the statistical soundness of the study. But why did they not find out more about this in their reporting? Perhaps some were so interested in the contrarian nature of the story–genomes aren’t all they’re cracked up to be–that they didn’t push hard enough to discover potential problems with the study.

One tip-off was the many stories that have been written questioning the value of commercial genome scans. Reporters should have asked whether the findings were new. That would not necessarily have uncovered the statistical issues, but it might have led reporters to scale back their coverage.”

 

Sober Second Thoughts?

Paul Raeburn, in the article cited above, concluded:

“If I had covered this story, I fear I, too, would have missed the issues that Hayden presents so clearly. The main lesson I can draw from this is that reporters ought to be as skeptical and vigilant as they can be, especially when writing about subjects, such as this one, that they have written about many times before–enough to have formed opinions that might be getting in their way.”

 I myself have posted before about the “good, the bad and the ugly” of public engagement in research. Science can never again be an ivory tower exercise – much of the research, including that study in question, is done at public expense, whether that be taxpayers’ dollars or charitable donations. The public has a right to be informed as to how their dollars are used, and researchers have a responsibility and accountability to inform them. Very often, it is science journalists, health reporters, broadcasters and the like who have a central and trusted role to play smack in the middle, as a critical conduit to the public to make sure that happens.

But they have to get it right if they are to hold that public trust.

Still, as Mr. Raeburn said, how many reporters have the qualifications and expertise to really dissect increasingly specialized science and increasingly complex data sets? I *HAVE* some qualifications and I certainly can NOT keep up with, nor even understand, much of the highly complex, jargon-filled science that even I try to write about.

So, while it is easy to criticize the reporters who may have rushed to judgment and perhaps overly sensationalized what for many is a non-story, on balance one surely also has to hold accountable the very scientists themselves who may have allowed this steamroller to roll down the hill, and indeed, from what I can surmise, may have even given it quite a little push to get it going downhill in the first place, whether intentionally or not.

"Personal" Genomics...Part 2

Rather than amend or append my post on "personal genomics" from earlier today, I thought I would provide a new one that expands on the "personal" genomics thread.

But this time, it is not about privacy, confidentiality or other kinds of ethical concerns, per se. Instead it goes right to the heart of the matter of "just how useful will this information be to the average person", average meaning a person at 'average' or 'slightly above average' risk for a disease.

Some details of a very interesting study were released today at the Annual Meeting of the American Association for Cancer Research (AACR) in Chicago (meeting website here), and published simultaneously in the journal Science Translational Medicine. Authored by a team at Johns Hopkins in Baltimore and led by world-famous researcher Dr. Bert Vogelstein, the study, entitled The Predictive Capacity of Personal Genome Sequencing suggests that the information that might be provided could actually be of limited usefulness.

Rather than me trying to paraphrase the whole thing, I refer you to an excellent article by Gina Kolata in the New York Times today (Study Says DNA’s Power to Predict Illness Is Limited) that offers an excellent recap.

For a second viewpoint, but same conclusions, you can check out this column from  Robert Bazell entitled "Gene tests: Your DNA blueprint may disappoint, scientists say" online also today at msnbc.com

Bottom line, it seems to me, is that we really have to be more careful than ever about exposing ourselves to privacy, confidentiality, insurabililty and other legal and ethical dilemmas, especially if the risks might outweigh the gains in many, if not most, cases.

Clearly there is no definitive pronouncement to be made one way or the other yet - it is far too early days for that. But it is good to have these debates with our eyes wide open.

Sure beats the alternative...

Join the Ontario Health Study

If you live in Ontario (Canada) then I encourage you to be an active research participant (and it's easy to do!) and be part of the Ontario Health Study (OHS). The OHS website is here.

Not limited to cancer, but to all chronic diseases, the Ontario Health Study is a huge research undertaking that takes community involvement to the limit. You can help!

In their own words:

The Ontario Health Study is an ongoing research study investigating risk factors that cause diseases like cancer, diabetes, heart disease, asthma and Alzheimer’s. If you live in Ontario and are 18 or older, you can take part in the OHS by filling out health-related questionnaires online. Researchers will use this health information to study how our lifestyle, environment and family history affect our health over time and to develop strategies for the prevention, early detection and treatment of diseases.

Personalized Medicine [Part 1] – The PROMISE...

Whether you call it personalized medicine, or precision medicine, or whatever, the promise of the $1000 (or less) genome has captured the imaginations and aspirations of the public and the research community alike.

This excitement is not ill-considered. One can assume that there are going to be vast improvements in our ability to prevent, diagnose, treat and cure cancers as we learn more and more about the mutations that drive the diseases.

As  Colin Hill contributed to Forbes on Feb 9, 2012 in his post “Beyond the $1000 Genome on the Forbes website.

“Larger availability of complete genomic data will have a profound near-term impact on cancer research. The ability to rapidly and economically sequence individual patient tumours will help us to better understand the biological mechanisms of cancer and will facilitate data-driven patient stratification. This, in turn, will facilitate more effective clinical trials and speed the development of new therapies.

The significant near-term growth of rich genomic data will impact the patient care side too. Companies ... will use this data to perform molecular analysis of tumours that will assist in pinpointing the optimal treatment strategies for individuals with cancer.”

But of course, no matter how cheaply it can be done, sequencing a single patient’s genome is not going to tell us much if we don't know what we are looking for. How will we know a “bad” mutation from a “neutral” mutation? As we saw with Craig Venter’s genome, his natural amount of genetic variability, and presumably yours and mine as well, is very high, and yet Craig Venter is by all accounts a healthy man.

Well, the supposition is that if we look at enough DNA sequences of enough cancer patients then a pattern will start to emerge and we will start to see certain mutations showing up over and over. How many will there be? Will some mutations be more directly linked to actual “causation” of the cancer (so called “driver” mutations) or will some be there as a consequence of the cancer and not actually involved in the origin or progression (so-called “passenger” mutations). Will we be able to tell the difference?

Answers to these crucial kinds of questions require a lot more data than we currently have. And that is where and how the International Cancer Genome Consortium (or ICGC) was formed.

Inaugural Meeting of Genome Scientists in Toronto 2007

Much like the Herculean world-wide effort to collaborate in determining the very first human reference genome sequence, the ICGC is also a massive consortium (website at http://icgc.org/) . The ICGC is a consortium not so much of scientists but of whole countries. The consortium was formed in 2008 after an inaugural meeting (Toronto) and report in 2007, to bring a global effort to bear in sequencing enough genomes for each of perhaps 50 or more types of important cancers so that we could start to answer some of the questions posed above. It is estimated that perhaps several hundred genome sequences derived from patients with any individual type of cancer may needed in order to be able to have statistical confidence of which mutations may indeed be the “drivers” vs. those that may simply be the “passengers”. If you consider the effort, and cost, of sequencing hundreds of genomes for each of perhaps 50 types of cancer, you start to see the enormity of the task, and why a consortium of countries in needed.

The ICGC is therefore funded in the main by governments and government agencies (federal and provincial here in Canada) of countries, along with some notable cancer charities and other funders. Each participating member country of the consortium is expected to invest at least $20 Million overall to that country’s activities. Furthermore, a commitment to fully, openly and quickly share ALL data with other qualified researchers world-wide is an absolute requirement for membership.

The secretariat of the ICGC is in Toronto, at the Ontario Institute for Cancer Research (OICR; http://oicr.on.ca), and Dr. Tom Hudson, an internationally renowned genomics researcher who is President and Scientific Director of the OICR heads the Executive.

OICR also operates the main data coordination centre for the consortium. Tom took me on a tour of the OICR server room about a year ago and I can tell you it is like something out of the movies :) The air conditioning costs alone for  just keeping the server room(s) cool must be enormous!

The goal of all of this of course is to have an international database of “signatures” of dozens of types of cancers, with enough confidence that we can start to use that data to better understand cancers at the gene and molecular levels, and be better able to determine predisposition to cancers (leading to better prevention strategies); to determine better and more pin-point diagnostic and marker mutations (leading to earlier diagnosis and better interventions), and to determine many new therapeutic targets for treating and curing more and more cancers.

In Canada, we have taken a leadership role for three different types of cancer – Pancreatic Cancer (ductal adenocarcinoma of the pancreas; collaborating and funding organizations can be found here; Brain Cancer (pediatric medulloblastoma; collaborating and funding organizations can be found here; and Prostate Cancer (adenocarcinoma of the prostate; collaborating and funding organizations can be found here.

 

Whether the ICGC actually achieves all of its lofty goals is of course yet to be fully seen. What is clear is that a major undertaking like this brings out what I consider to be the very best in science and scientists: the desire and willingness to not compete but instead to openly collaborate, to share data, and to work for the common good in ways that no single researcher or even a single country could manage.

This is so-called “big science” at its best and we should all be pleased that it is being undertaken in the international arena, and in the collegial and cooperative manner that it is.

Welcome to the World of Genomics... [Part Three]

In the last post [World of Genomics Part 2] I tried to give you a flavor of just how far and how fast the science of genomics has progressed in the last decade. We are truly getting to a point where an individual human DNA sequence will be available for $1000 or less.

There is also increasing prospect of what is called “direct to consumer" products where you will be able to order your genome from a company on the Internet without any intervention by your physician. In fact, I suspect this is already happening in limited ways but will pick up steam very soon as the affordability continues to drop.

As much as the technology will afford us an unprecedented technological advance into our understanding of human diseases such as cancer, it is my belief that this era of “cheap” genomes is also ushering in some unprecedented questions of ethics and law that we are not yet facing head on, and need to start debating and discussing as a society ASAP.

Top of the list perhaps are issues of privacy and confidentiality. 

Where is your genome sequence going to be stored? I could imagine an app on your smart phone in the not so distant future! Do we really want to have our genomes floating around in cyberspace? Do we trust some central database (e.g. a government database) to house this information? I will wager that many of you already feel uncomfortable about the fact that the CRA in Canada or the IRS in the United States holds so much informational power over you by having your detailed tax records and related files in their databases. I cannot imagine a piece of information more personal or more confidential than my own detailed DNA sequence; will I really trust that it will be kept secure on the Internet or in someone's file cabinet?

The privacy and confidentiality issues lead us then to questions such as insurability. Suppose an individual is carrying seven particular mutations that might, and I stress might, predispose them to a particular disease. And suppose that information is now made available to an insurance company, and as a result life insurance or mortgage insurance or some other form of insurance is denied because the risk is deemed to be unacceptably high? What happens then?

What, in fact, does “predisposition to disease" really mean anyway? In the vast majority of cases this is not a clarion signal that the disease will develop. It merely says that you MIGHT need to take different levels of precaution than your neighbour in order to prevent or avoid that disease from occurring in the first place.

And if your doctor is able to determine from your mutational status that you have a predisposition to some particular disease, what about your “need to know” vs. your “right to know”? In some jurisdictions, such as France, the obligation of a physician to disclose this information is enshrined in law, as I understand it. In the US and in Canada there are no such regulations yet. Who is going to make the decision about when your health care professional should, or must, advise you of your mutational status, especially if it doesn't actually mean anything finite in the immediate sense of the word? If there is nothing you can do, then how important is it for you to know? Is it your right to know?

And even if it is your right to know, is it possible that we will end up creating so much anxiety and stress in individuals who learn of a particular mutational status that we will in effect “stress” them into the very diseases we are trying to prevent? The notion of creating so many self-fulfilling prophecies is very real in my view .

And then there are issues of economics and policy. The better able we are to define specific sets of mutations and to tailor treatments to those sets of mutations, it could be imagined that we will need more and more targeted drugs. While targeting and specificity are a good thing, most of these drugs are not cheap! One could rightly ask why would we be developing more and more expensive drugs when we can't even afford the ones that we have now.... 

And how will decisions be made about who has access to which drugs? We already see significant differences in Canada from province to province about cancer drugs that are paid for by the public health system in one province but are not available to patients in a neighboring province.

And from a policy maker's point of view, it could be fair to ask "how much is X months of someone's life worth?" If an expensive drug can prolong a cancer patient's life by six months, for example, who makes the decision about "at what cost"? I can easily see that if the patient in question is your mother, or your son, or your sister etc. then you might justifiably argue that ANY cost is worth it - you are prolonging the life of a loved one.

But if you are the Minister of Health and you have to look at this in terms of benefit vs. cost to society at large, you no doubt will need to look at this more objectively and dispassionately.

The answers to these kinds of questions will come from new kinds of cancer research, but it won’t be in the usual laboratory settings. Instead, we need to accelerate our efforts into research in:

  

•  health economics of cancer
• how will money be best spent?
• health services research
• how will services be best organized?)
• health policy research
• how will information be provided to policy makers for best use?)
• ethics research
• how will resources and access to service be maintained in the fairest way for all patients? 
• how will we protect vital personal and confidential information? 
• who owns the data? 
• who defines a patient's “need to know” vs. "right to know”?)  

The point is, that we are at a stage in the development of very powerful technologies that are going to create opportunities but also some very fundamental ethical issues that I do not believe we are ready to deal with at the societal level.

There have been a few other technological “tsunamis” that have broken on society and changed our world irrevocably in the past. One of these was of course the advent of nuclear technology and all of the good and ill that brought with it. 

Another was the development of recombinant DNA technologies that brought with it the modern era of molecular biology, of which these genome science opportunities are the latest wave. I am not old enough to know what sort of public consultations, if any, accompanied our ushering in of the nuclear era, but I do very much remember some of the public debates that happened in the early 70’s around the advent of the new molecular biology.

Without in any way suggesting that the outcomes of those debates and consultations were appropriate or not, at least an attempt was made to engage the public and to inform people of what was coming, and to attempt to assess it from both a benefit and risk perspective. I don't see the same level of engagement happening now with the new genome technologies and I think it is overdue. 

These issues are too important and the ramifications are too far-reaching to not have these debates and discussions right now…