The 2nd U.S. Circuit Court of Appeals in New York said the settlement, calling for payments of as much as $18 million, was unfair because it shortchanged authors who did not register copyrights in their works. These authors represented more than 99 percent of the claims covered by the 2005 settlement.
Wednesday's 2-1 decision followed a unanimous March 2010 U.S. Supreme Court decision, Reed Elsevier v. Muchnick, that resurrected a possible settlement, which the 2nd Circuit earlier threw out on the grounds that it lacked jurisdiction.
Publishers in the lawsuit included Reed Elsevier, New York Times Co, News Corp's Dow Jones & Co, Thomson Reuters Corp and Knight Ridder, which was bought by McClatchy Co in 2006.
The settlement, reached through mediation, came four years after the Supreme Court in 2001 said publishers violate copyright law when they reproduce freelance works electronically without first obtaining permission from copyright owners.
A group of 10 authors objected. They contended that authors who had not registered their works could see their recoveries unfairly reduced, and that the settlement freed publishers from potential litigation over too many claims.
Writing for the appeals court majority, Circuit Judge John Walker said a lower court judge erred in combining this group of authors with other groups, who could recover more on their claims, into a single settlement.
"Although all class members share an interest in maximizing the collective recovery," Walker wrote, "their interests diverge as to the distribution of that recovery because each category of claim is of different strength and therefore commands a different settlement value."
He said the "simplest and most logical approach" might be to create classes for each type of claim, with different lawyers for each class. The 2nd Circuit returned the case to the U.S. District Court in Manhattan for further proceedings.
Circuit Judge Chester Straub dissented in part, finding that class-action certification was proper.
"The defendants are very disappointed," said Charles Sims, a partner at Proskauer Rose representing the publishers. "The decision tells authors they will have to wait even longer for any money." He said the publishers are considering their options.
Charles Chalmers, a lawyer for the objecting authors, said his clients are "gratified" with the decision, and "look forward to a revised resolution that insures protection for the 99 percent of freelance articles involved in the action."
A lawyer for the remaining authors did not respond to a request for comment.
One of the mediators in the 2005 settlement was Kenneth Feinberg, who oversees a $20 billion fund that BP Plc set up for victims of last year's Gulf of Mexico oil spill.
Jonathan Tasini, the lead plaintiff in the 2001 Supreme Court case, in April sued AOL Inc and The Huffington Post, saying they failed to pay bloggers for their work.