ONE DAY AFTER DEPARTMENT OF HEALTH AND HUMAN SERVICES ANNOUNCED ITS NURSING HOME QUALITY INITIATIVE, THE GAO ISSUED A HIGHLY CRITICAL REPORT THAT RECOMMENDED DELAYING THE INITIATIVE
Health and Human Services Secretary Tommy Thompson announced nationwide implementation of the Nursing Home Quality Initiative on November 12, 2002. The Initiative has been operating as a pilot project since April 2002. The two-part Initiative involves public reporting of "quality indicators" or "performance measures" (data derived from facility-reported resident assessment information) and technical assistance to nursing facilities by Quality Improvement Organizations.
Expressing serious reservations about (1) the accuracy of the minimum data set (MDS) data used to calculate the performance measures, (2) the appropriateness of the quality indicators identified for public release, (3) the "potential for public confusion" over contradictory information and inconsistencies between quality indicator scores and deficiencies identified in surveys, and (4) inadequate preparation by Medicare and QIO staff to answer telephone calls from the public about the Nursing Home Quality Initiative, the General Accounting Office recommended that
. . . the Administrator of CMS delay the national reporting of quality indicators until (1) there is greater assurance that quality indicators are appropriate and based on accurate data and (2) a more thorough evaluation of the pilot is completed.
GAO, Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National Implementation Is Premature 4, GAO-03-187 (Oct. 2002).
In comments on the draft report dated October 22, CMS Administrator Thomas A. Scully rejected the GAO’s recommendation:
We welcome GAO’s report, because the issues raised make us more convinced than ever that moving forward to launch the Nursing Home Quality Initiative (NHQI) is the right thing to do. We believe the quality measurement information we are sharing is reliable, valid, accurate, and useful. Waiting for more reliability, more validity, more accuracy, and more usefulness will delay needed public accountability, and deprive consumers, clinicians, and providers of important information they can use now. Quality of care in nursing homes will improve more quickly through the positive stimulus of this initiative now.
Memorandum from Thomas A. Scully to Kathryn G. Allen, Director, Health Care – Medicaid and Private Health Insurance Issues (Oct. 22, 2002), reprinted in GAO report, at page 28.
The GAO found
Overall, CMS’s initiative to augment existing public data on nursing home quality has considerable merit but its plan for nationwide implementation in November 2002 is premature. Conceptually, CMS’s plan encourages consumers making a decision about a nursing home to consider those with positive quality indicator scores – a use of market forces to encourage poorly performing homes to improve quality of care or face the loss of revenue. Such a plan hinges, in part, on appropriate quality indicators that consistently distinguish between good and poor care provided by nursing homes. However, CMS has not yet adequately resolved a number of open issues regarding the appropriateness of the quality indicators selected for public reporting and the accuracy of the underlying data.
Id. 3.
The GAO first questioned the validity and reliability of the MDS data and quality indicators, asking whether there has been sufficient validation of the indicators. Id. 7. The GAO was critical of the limited "available portions" of the Abt evaluation of the indicators. Abt’s validation study used a sample from six states and the facilities in the evaluation were not necessarily representative of even those states – facilities were allowed to decline to participate in the evaluation and more than 50% did decline. Id. 8. Moreover, the GAO could not determine from the information provided how Abt validated the indicators. "Unresolved questions" also remained about the risk adjustments that were used. The GAO noted that data released by CMS in March 2002 "demonstrated that Abt’s risk adjustment approaches could either lower or raise facility scores by 40 percent or more." Id. 9. The GAO expressed concern that CMS was not waiting for its expert advice through the National Quality Forum process. Although NQF’s Steering Committee planned to make final recommendations about the indicators to be used in the national NHQI in August 2002, CMS asked NQF in June to delay making its recommendations until 2003. Id. Finally, the GAO noted that in the absence of consensus on risk adjustment of the indicators, CMS was reporting two indicators in two ways – with and without risk adjustment. Such public reporting would likely cause confusion for consumers, the GAO reported. Id. 10.
The GAO next questioned the underlying accuracy of the MDS data. Id. Although Abt reported in August 2002 that the MDS data used to calculate the indicators were reliable, a February 2001 report by Abt questioned the reliability of the same data. Id. The earlier Abt report found error rates for the MDS were high, particularly for some of the MDS items that are now being used to report quality indicators. For example, error rates for identifying pressure sores was 18% and for pain intensity, 42%. Id. 12. The GAO cited a study by the Inspector General that also identified differences between the MDS and the medical records. Id. State survey agency officials in three of the pilot states expressed concern to the GAO "that the public reporting of quality indicators may lead to underreporting of certain problem areas, such as pain management." Id. 12, note 19. The GAO noted that CMS is "in the process of implementing a national MDS accuracy review program expected to become fully operational in 2003." Id. 10-11. In an earlier report, the GAO criticized CMS’ review programs as too limited in scope (GAO, Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should Complement State Activities, GAO-02-297 (Feb. 2002)). Id. 11 and note 14.
Third, the GAO expressed concerns that the public might be confused by the data. Id. 15. The Initiative’s correlation of a high score with poor performance is "counterintuitive and could prove confusing to consumers." Id. In addition, reporting actual scores, rather than a range of scores (such as low, medium, and high), "implies a confidence in the precision of the results that is currently a goal rather than a reality." Id. 15-16. Consumer confusion could also result from efforts to interpret missing information. Six percent of facilities in the six pilot states had no score for any of the nine quality indicators; for individual indicators, 9% to 40% of facilities were missing scores. Id. 16. The GAO also found that the scores sent "conflicting messages." Id. 17. Seventeen percent of facilities in the pilot states "had an equal number of highly positive and highly negative quality indicator scores." Id. Inconsistencies between quality indicator scores and survey deficiency data also create confusion:
For example, 17 percent of nursing homes with four or more highly positive quality indicator scores and no highly negative scores – seemingly "good" nursing homes – had at least one serious quality-of-care deficiency on a recent state survey. We have found that serious deficiencies cited by state nursing home surveyors were generally warranted and indeed reflected instances of documented actual harm to nursing home residents. Moreover, 73 percent of nursing homes with four or more highly negative quality indicator scores – seemingly "bad" facilities – had no serious quality-of-care deficiencies on a recent survey . . . . The latter situation is consistent with our past work that surveyors often miss serious quality-of-care problems. Nevertheless, consumers will generally lack such insights on the reliability of state surveys that would permit them to better assess the available data on quality of care.
Id. 17-18.
Finally, the GAO found that "CMS is not prepared to respond to consumers’ questions." Id. 15. Scripts used by the Medicare hotline and QIOs "did not address the issue of responding to questions about conflicting or confusing quality data." Id. 18-19. When the GAO called the Medicare hotline and QIO toll-free numbers, "staff were not adequately prepared to handle basic questions about the quality data available under the pilot." Id. 19. The GAO found, "In general, QIO staff were not prepared to respond to consumer questions." Id. Only two of the staff at the Medicare hotline and QIOs were "generally knowledgeable about different types of quality data." Id. "[O]thers were unable to answer simple questions and the majority provided erroneous or misleading data." Id.
The GAO also reported that CMS’ evaluation of the pilot is "limited and will not be completed prior to the national reporting of quality indicators." Id. 20.
According to CMS officials, the pilot evaluation was never intended to help decide whether the initiative should be implemented nationally or to measure the impact on nursing home quality. While CMS is interested in whether nursing home quality actually improves as a result of the initiative, it will be some time before such a determination can be made. Thus, CMS focused the pilot evaluation on identifying improvements that could be incorporated into the initiative’s design prior to the scheduled national implementation in November 2002.
Id. CMS’ initial pilot evaluation was available in October 2002; CMS expected final results in 2003. Id.
The GAO recommended that CMS delay implementation of the initiative until
Id. 22.
The report also responds at length to CMS’ comments. Id. 22-26.
The report was prepared at the request of Senators Charles A. Grassley and Christopher S. Bond.
The GAO report (26 pages plus appendices) is at http://www.gao.gov/new.items/d03187.pdf. Senator Grassley’s statement on the Initiative is at http://www.grassley.senate.gov/releases/2002/p02r11-12.htm.
© Center for Medicare Advocacy, Inc. 05/02/2008