It all started when I read
this post at James Dwight's
Soxblog, also linked over at the right. Dwight dissected a
New Yorker article by one
Dr. Atul Gawande, in which Gawande asserted that the performance of various cystic fibrosis treatment centers could be plotted on a bell curve, and that the variation in performance was due solely to the difference in provider abilities across the centers. Dwight took exception both to Gawande's assumption and sloppy methods, and posted about it. I found Dwight's posts interesting but didn't really give it much more thought.
Some time later,
Virginia Postrel came to praise this same article, in her post
Medical Kaizen.
In general, I like the way Postrel thinks, so I was surprised that she was ignoring the deleterious effects the Gawande article could have, along with the shoddy analysis he had undertaken. So I (along with other readers, apparently) emailed her links to Dwight's posts.
Postrel's husband and Dwight had an email exchange in which they batted around a few topics.
Dwight's
last post mentioning this topic reprinted a letter to the
New Yorker which re-affirmed some of his points -- yay! I was pleased that the dialog came to this conclusion.
So today, I finally made it back over to Dynamist (Postrel's blog) after a couple of days away, and there were a few final Gawande-related posts.
Here's a copy of Dwight's correspondance with Professor Postrel. My reading is that the Professor was not swayed in the least by Dwight's arguments, whereas Dwight was more than willing to concede the points which were debatable.
Here is the final post on the topic. This last post links to
Jim Hu's blog, and he has posted several times on the topic, as well.
None of these bloggers have comments enabled, but both Dwight and Postrel accept email. Hu, on the other hand, will only accept comments via blog-posting and trackbacks(!). (Maybe that's what finally prompted me to write up something on this myself.)
Having read the original Gawande article and all the relevant blog postings, and ruminating on this topic for quite a while, I find myself squarely in Dwight's corner, and piqued enough to actually want to post on it.
Working from last to first, Hu makes assumptions about Dwight that I would not make, but then again, I've been reading Dwight for months now and consider him someone I 'know' as well as possible via the blogosphere. So when Dwight says, "you'll have to trust me on this," well, I do. Hu doesn't, and questions Dwight's bona fides to be commenting on CF. That was one of the things that bothered me about Hu, and also about the Postrels: they all pretty much dismissed Dwight's objections to Gawande's article, out of hand. Hu also took the trouble to nit-pick to a truly silly extent Dwight's description of when you would, and would not, find bell curves. Seriously, most people don't describe their height using a +/- margin of error.
Technically what Hu writes is correct, but c'mon, dude! Give us all a break!
The Postrels keep harking back to The Professor's experience in other industries:
There is a vast body of empirical evidence in fields ranging from computer programming to automobile manufacturing that performance variations among similar units are large and persistent. To which I must point out, there are many fewer variables involved in software development and car manufacturing, and therefore it is easier to collect such empirical evidence in a way in which you can actually compare results from one center to another. I'm basically restating Dwight's argument here, but for the record: comparing outcomes of any two medical establishments without first normalizing the patient pool and controlling for other external factors that will effect the outcome being measured is
extremely difficult. Gawande used the CFF data because, basically, it exists, but I still think he (Gawande) is guilty of
torturing the data until it confesses. (1)
Why do I care about this? Because I'm surprised that so many obviously intelligent people failed to read this article with the kind of critical eye that it requires. I'd like to remind both the Postrels and Hu that
the plural of anecdote is not data.(2) Yet Prof Postrel quotes Gawande's anecdotes regarding patient care in both the Minneapolis and Cincinnati centers as evidence of the superior dedication to achieving patient compliance in MN.
How about this statement from Gawande, in discussing the success of a maverick treatment provider:
He had not had a single death among patients younger than six in at least five years. My immediate reaction to this statement?
How many patients younger than six has he treated in the last 5 years? Without the answer to my question, there's no way to evaluate Gawande's statement. The entire article is rife with things like this: blanket statements that I'm sure are true, but that conceal as much as they reveal.
Gawande started out with the premise that the doctors must be to blame, because "there is a bell curve" of outcomes, and it has to be the doctor's "fault", right? But he himself wrote:
The hundred and seventeen CF centers across the country are all ultra-specialized, undergo a rigorous certification process, and have lots of experience in caring for people with CF. They all follow the same detailed guidelines for CF treatment. They all participate in research trials to figure out new and better treatments.
It seems to me that Gawande is admitting that the CFF has already implemented standard "best practices." He continues:
You would think, therefore, that their results would be much the same. Yet the differences are enormous.
Instead of asking, "what are the factors contributing to the different results," Gawande takes his pet theory and forces it to explain what's going on. And he's not content to do only that, either:
Patients have not known this. So what happens when they find out?
He has to go and push his incomplete analysis on the patient community, without apparent thought for the consequences.
Dwight and Postrel discussed the stability of the patient population, and how that would (or would not) affect any rigorous analysis of outcomes. I wish that Gawande had looked at the length of
staff assignments. Do doctors specializing in CF come to a particular center and spend their entire careers there? Some may, some may not -- I have absolutely no idea,
because Gawande never told me. Having taken the time to lay out the scenario -- hey, there's a bell curve here, and Cincinnati's below average! -- he then goes on to blame the staff,
without telling us how long the doctors actually practiced in Cincinnati. Maybe Cincinnati is a teaching center, and you see the difference in outcomes because the staff rotates through, and the cumulative effect is of a less-experienced provider. Or maybe (very likely, in fact, from the anecdotal evidence Gawande did provide), there are one or two truly brilliant providers who have been working in MN for decades, who have had a disproportionate effect on the outcomes at the MN center.
I'm still trying to figure out why this kerfuffle has pushed my buttons. I think it's because I've spent so much time entangled with the medical community over the past 4 years or so. I've had good doctors and bad doctors... at last count I think I have something like 10 or 11, which seems like way too many to me. I'm keenly aware of many of the problems we're all facing in healthcare these days, particularly in trying to extract the best service and care out of the providers we are dealt from our insurance companies' directories.
It's apparent that the Postrels, and Hu, think that Gawande's intent was to subject the medical industry to analysis and criticism in an effort to improve it. There was that recent
NYTimes article talking about pneumonia vaccines and improving patient care, blah blah blah "best practices", after all.
I guess I'm not feeling that charitable towards Gawande; like Dwight, I think Gawande is irresponsible. You can't pin the blame on the doctors for the range of outcomes until you've rigorously analyzed the data, and it is quite clear that Gawande did not do that. If Gawande was seeking to inspire the practitioners on the bottom half of the curve (the "underperformers") to move up, why did he have to involve the patient community? Why does he ask,
[W]hat do you do when the research shows that patients are getting care of widely different quality, when that isn't
necessarily what the research is showing?
There's far too much Gawande-ego and too many unfound assertions, and far too little analysis and critical reasoning, here. Gawande states:
Once we acknowledge that, no matter how much we improve our average, the bell curve isn’t going away, we’re left with all sorts of questions.
But Oren Grad, known to Postrel as knowledgeable on medical outcomes research, refuses to give Gawande any such acknowledgement. He
wrote:
I do think Gawande, who in general is quite good at writing about medical topics for general audiences, went astray this time in hanging the article on the implications of the bell curve.
Another way of looking at [it ...] is to think of it as pushing the entire performance curve up and reducing its spread so that even performers who are, in a mathematical sense, below average nonetheless deliver a level of quality that is entirely satisfactory in a substantive sense. If they succeed, then why should anybody care who is below average?
I think Grad has hit the nail on the head, here: even if there is a distribution of outcomes, that does not necessarily imply that "below average" is the same as "unsatisfactory", which is Gawande's key unwritten assumption.
The Postrels seem to think it's enough that Gawande is asking the questions he does, about quality of care, best practices, patients seeking out better performing providers, etc, and what the implications of all these things could be. I suppose I could be throwing out the good with the bad by having such a generally negative reaction, here. Is it a good thing that Gawande wrote the article? Aren't these issues important, shouldn't they be discussed? Yes, I think these are important issues and believe me, like any chronically ill person, I'm about as interested in health care as you can be when you're not employed in the industry. But I don't think Gawande's article does much to advance the topics that should be discussed, since he keeps harking back to his bell curve.
Quite simply, I'm not buying it, and I won't buy it until I see much more rigorous analysis. My own experience has demonstrated repeatedly that the way to get the best care is to manage your own case as aggressively as possible. Obviously this is not ideal, but there ya go. Would implementing Gawande's report cards and best practices do much to improve anything? I doubt it, given the entrenched nature of the medical establishment and the difficulty in measuring performance so as to provide meaningful "grades". The fastest route to improved medical care is through well-educated patients.
Perhaps what we really need is a new class of professional patient advocates who can mediate between the medical professionals and their patients. Very few patients can be their own advocates; most are intimidated by the jargon and afraid of hearing unwelcome news. Still, if there was someone around who could help the Everyman navigate the churning waters of the medical industry and insure that
best practices are followed, it might even save the insurance companies a bundle of money. Enough money to pay the Patient Advocates a decent salary, for example.
I don't know if this idea is original or even new, but I like it. Even though I feel like I'm on top of my diagnoses and treatments for the most part, somedays I'd just like to be able to run it all by someone and say, "what do you think? am I getting the care I need?" I have no way of answering that question other than by the very subjective standard of how I'm feeling on any particular day. Frankly, that's not good enough: in the summer and early fall, I felt pretty darn great, only to find out that there was cancer metatastizing in my neck. Now I feel rather icky but there's a good chance I'm cancer-free... but I won't know for several months, when we do the next whole body scan.
Getting back to Gawande's thesis about doctor performance, patients
should fire their doctors when they're "underperforming", but there's no need to go looking for a bell curve or to implement a grading system to place our doctors on it. You can make that "performing/under-performing" assessment based on a single data point: how your doctor treats
you. Well-educated patients have enough information to be able to make that decision themselves. Patients need to understand that this is indeed their responsibility! When a practitioner has been "fired" enough times because he has failed his patients, he'll be forced to re-evaluate his methods and bring them up to an acceptable standard.
Gawande's grading and bell curves wouldn't improve medical care. The complexity involved in the grading, the transient nature of practitioners and patients, and a host of other factors would make it all too easy to dismiss any particular "grade", or all of them, in fact. Doctors and treatment centers would resent having these measurements imposed on them, patient populations would routinely misinterpret them, and havoc would generally result. Far better to approach this problem at the individual level, and see how the incremental changes add up, over time, to greatly improved service... and possibly, improved outcomes, as well.
-----
(1) I first saw this written on a blackboard (yes, I am that old) when I was a freak undergrad at the Sloan School.
(2) I have no idea who said this first, either. It wasn't me.
