In a decade-old case that revolves largely around how to interpret surgical outcome statistics, an appellate panel has ruled that a defamation lawsuit against CNN and its Anderson Cooper 360° program should go to trial.

A circuit judge had issued a summary judgment in CNN’s favor in 2023, effectively throwing out the case filed by pediatric surgeon Michael D. Black. A three-judge panel from Florida’s Fourth District Court of Appeal disagreed in September, noting that “a reasonable jury could return a verdict for Black,” but declining to express an opinion on the case’s merits.

The lawsuit stems from a 2015 series of CNN stories about St. Mary’s Medical Center in West Palm Beach, Florida, that reported the hospital’s pediatric cardiothoracic surgery program had a death rate more than three times the national average. Between 2011 and 2013, 6 of the 48 infants who underwent cardiac surgery there died — numbers that CNN said represented a 12.5% death rate. The Society of Thoracic Surgeons reported at the time that the national death rate for pediatric cardiac surgeries was 3.3%.

St. Mary’s closed its program 2 months after the stories aired. The hospital subsequently paid $102 million to settle nine family lawsuits filed against it. But it maintained, without releasing the basis for its conclusion, that its risk-adjusted mortality rate for the 4 years ending June 30, 2014, was 5.3%. The medical center criticized CNN’s “flawed analysis,” which it said “mischaracterizes program performance.”

Before CNN aired its series, St. Mary’s CEO sent a letter calling CNN’s data “inaccurate” because it did not include “all cases and/or procedures performed as part of the program.” Declining to release any data, the letter noted that many programs across the country do not reveal “raw mortality data because such data does not give proper context for the complexity and severity of each case, which could potentially lead to providing misleading information to consumers.” Instead, the letter said, St. Mary’s had submitted data to the Society of Thoracic Surgeons, which gave St. Mary’s “a two-star ranking,” the same ranking the society gave nearly two thirds of similar programs in the US.

Dispute Over Data: Raw vs Risk-Adjusted

The debate over raw vs risk-adjusted data has raged for decades, noted Albert Wu, director of the Center for Health Services and Outcomes Research at Johns Hopkins Bloomberg School of Public Health, Baltimore.

Raw data about deaths as a proportion of total surgeries “is very difficult to interpret without some context, whether it be qualitative or quantitative,” he said. “When presented with those statistics, a natural and understandable rebuttal from doctors can be: My patients are sicker. And in some cases, that may be the case.”

That’s where risk adjustment comes in.

“There has been a terrific amount of work on this dating back to the 1980s, particularly in New York and Pennsylvania, which created detailed registries of all cardiac surgeries,” he said. Published annually, the reports note both raw and risk-adjusted mortality rates for every cardiac surgeon in the state.

“Risk adjustment isn’t perfect, but it certainly adds important information,” he said. “If risk-adjusted mortality rates were never calculated officially, there would be a risk that some surgeons might cherry-pick their patients and selectively operate only on people at very low risk so that their death rates would appear lower.”

Small Sample Sizes, Big Questions

Even when adjusting for risk, however, it’s difficult to analyze a program when there are very few surgeries in the dataset. Noting that some surgeons operate 300 times in a year, Wu said that Black’s 48 cases over the course of 2-3 years pose a challenge.

Nonetheless, raw numbers are a critical start to the story, said statistician Arlene Ash, who is chief of the Division of Biostatistics and Health Services Research in the Department of Population and Quantitative Health Sciences at the University of Massachusetts Chan Medical School, Worcester, Massachusetts.

“There’s no excuse for failing to give the raw numbers,” said Ash, who consulted with CNN’s reporters as they prepared their stories. “The raw numbers may not tell the whole story. They may actually mislead you into thinking that something is fine when it isn’t, or vice versa, but they are the basis from which we work. That said, you should never stop at the raw numbers.”

Risk adjustment factors in the case mix between straightforward and riskier surgeries, she said. “Every doctor I know will start by saying that their patients are more difficult than other people’s patients,” she said. “Risk adjustment tries to neutralize that argument.”

As Wu noted, the sample size can make a big difference, she said.

“These statistical tests and statistical adjustments contain random noise, especially when the sample you’re looking at is rather small,” she said. At that point, “it isn’t just the technical calculation. You need to think about what it means in context.”

This contextual information can include such things as how new a program is, the overall experience level of individual surgeons, and whether there are enough surgeries taking place to provide a broad mix of cases.

“You try to put into your formulas all the information that you can find that seems relevant to distinguishing between cases with a high risk of mortality and those with lower risks,” Ash said. “With these really sick babies, these neonates that are born with very serious heart problems, it’s not unexpected that some of them will die. But did CNN publish things they knew were misleading or wrong? No.”

She pointed to the fact that the network also reported on a Florida Cardiac Technical Advisory Panel’s conclusion that St. Mary’s lacked the resources to perform vital tests and services for pediatric heart patients. One member of that panel noted that the program performed too few surgeries to develop the requisite level of competence and expertise. Ultimately, the panel recommended that St. Mary’s stop performing complex cardiac surgeries on children.

Transparency vs Context in Public Reporting

When it comes to risk adjustment, Ash said, “You’re limited by the data you have.”

“The universe of all the kids who underwent this kind of surgery across the whole country was quite large,” she said. “But what was being asked was: Is the record of these roughly 70 cases at this one facility enough to be worried about?”