Beneficiary Satisfaction Measure is Not Used to It's Fullest Extent
Unlike report cards that use clinical measures, The Consumer Assessment of health care Providers and Systems (CAHPS) reports patient ratings of their experience with care. The CAHPS surveys were designed to provider consumers of health care with more information and to enable the consumer to make better choices about health care coverage. Many resources have gone into the development and administration of CAHPS surveys, and as a result, many reports and studies have been conducted on some of the effects of CAHPS. This report seeks to look at the literature surrounding the CAHPS program and address beneficiary issues, including whether the CAHPS survey accurately reflects patient experiences and whether the CAHPS survey is useful and beneficial for consumers making health care decisions.
Overview of CAHPS
In 1995, the Agency for health care Research and Quality initiated CAHPS in order to develop survey and reporting products to give consumers feedback on health plans and provider performance. AHRQ, through the CAHPS program, seeks to support the assessments of consumer experiences with health care. CAHPS was originally launched in response to concerns that there was little information available about health plan enrollees' own perspective on the quality of their plans. The role of CAHPS has been expanded, and now seeks to address needs of health care consumers, purchasers, health plans, providers, and policymakers. The current goals of the CAHPS program are to develop an effective patient questionnaire for comparing data, and to generate tools and resources for sponsors to produce understandable comparative information that is useful for consumers and health care providers.
CAHPS continues to evolve, and is now in its third stage:
CAHPS I: 1996-2001
- CAHPS began when AHRQ gave 3 cooperative agreements to research organizations that collaborated with AHRQ, CMS and others to develop test instruments for health plans.
- During this period, CAHPS becomes the de facto standard for measuring consumer experience with health plans.
- At the end of this period, CAHPS products achieved broad use. CMS, the National Committee for Quality Assurance, and most state Medicaid programs adopted the CAHPS health plan survey.
CAHPS II: 2002-2007
- The second phase of CAHPS addressed the growing demand for new surveys in other health care sectors. CMS requested collaboration on surveys for hospitals, nursing homes, in-center hemodialysis facilities, Clinician/Group Surveys, and others.
- Researches also sought to develop tools for survey providers to help in quality improvements.
- Many stakeholders realize the value of CAHPS as a quality improvement tool.
- During this phase, CAHPS was translated into Spanish and research was conducted on the cultural competency of the survey instruments.
CAHPS III: 2007-
- The emphasis of CAHPS is no longer simply on developing new surveys. The goals have shifted to refining and supporting CAHPS surveys, and increasing the use of CAHPS surveys.
Does the CAHPS Survey Accurately Reflect Patient Experiences?
Use of Proxies
Some respondents are not able to complete the CAHPS surveys themselves. These respondents may have vision problems or may not understand the language, or they may not be able to cognitively understand the survey questions. For various reasons, respondents may require the assistance of a proxy. When a proxy actually completes the survey for someone, they are called a proxy respondent, whereas proxy assistants help the respondent by either reading the survey, translating, or writing responses.
Proxy respondents have an effect on survey outcomes. Because of the potential for bias, AHRQ advises consumers to complete their own surveys. "To ensure that the responses to a CAHPS survey reflect the respondent's own experiences with care, consumers should complete their own surveys. It is especially important that providers not assist their patients because of the possibility of bias." AHRQ recognizes that when respondents get assistance from providers, they tend to respond more favorably. In order to enable data collection from people unable to read the survey or write the responses, AHRQ suggests administering the survey by telephone or providing translators. Information on the CAHPS website and in the studies makes the questionable assumption that the CAHPS surveys are administered fairly and without any influence from the providers acting as a proxy.
Medicare health plan surveys are the exception to the restrictions against proxy assistance. In Medicare health plan surveys, family or friends are permitted to assist beneficiaries who are unable to provide responses independently. In studies that have been completed on Medicare CAHPS, it has been found that proxy respondent differences are small for objective reporting items but larger for global ratings. Proxy assistance effects are small. The relationship of the proxy to the intended respondent also affects the proxy responses. Spouses' responses are found to be closer to the patients' intentions. Proxy respondents (non-provider respondents) typically give less positive evaluations then both the self-reporting respondents and spouse proxies. Data also suggests that proxies who assist the beneficiary may also influence responses, but more research is needed to see how the proxy influences global ratings.
CAHPS surveys that use proxies use a case-mix variable to analyze the proxy assisted surveys with the independently completed surveys. Using proxy respondents is important to ensure that the experiences of the most vulnerable patient groups are not omitted. However, CAHPS surveys need to be aware of the effect proxy respondents may have on the results. Studies have found that using case mix adjustment and propensity score rating or matching can improve the accuracy of surveys using proxy respondents. Additionally, when surveying a population where a high proportion of respondents will need proxy respondents or assistance, a good survey would place a greater reliance on specific and objective reports rather than on subjective global ratings.
One report by the Health Research and Educational Trust looked at the self-reported data, and concluded that people tend to rate themselves more favorably than is the case. This report discussed biases that people have about their own health and self-identity, and the tendency of people to downplay negative experiences. This study was not discussing the CAHPS survey specifically, and it also was not advocating a dismissal of self-reported data. Rather, this study brought to light factors that affect the accuracy of self-reported data and proxy data. This study was one of the few that discusses the psychological factors that can affect the outcomes of health surveys.
AHRQ's response to those who question the value of patient experience data is that patients are the most knowledgeable informants about their experiences with care. AHRQ distinguishes between clinical data and CAHPS surveys. "Until recently, patients' views of their care were given weight only to the extent that they correlated with clinical measures. Now there is growing acceptance of patients' perspectives on care as a valid measure in itself." CAHPS is designed to measure patient experiences; it is not substitute for a clinical measure of quality. Indeed, research has shown that self-reported data does not always reflect the clinical measures of quality. Therefore, CAHPS data is best considered distinct from and different than quality measures. When a consumer uses CAHPS data to make health care decisions, it will be important that the consumer is aware of the distinction between clinical measures and patient experiences.
Cultural Competence of Surveys
CAHPS surveys are distributed to a myriad of people who have a variety of educational levels, ethnic backgrounds, ages, and medical conditions, and who speak a variety of languages. Studies have used CAHPS data to find that racial, ethnic and linguistic minorities face barriers in access to care, and lower quality of care than non-minorities in the same health plans. These studies call for health plans to provide a more diverse workforce with training in cultural competency, extended clinic hours, and interpreter services.
Most studies discussing ethnicity and language focus on the results of the CAHPS surveys, but they also tend to conclude with a call to construct surveys that are sensitive to different cultures and languages. A study by the Journal of General Internal Medicine states that "[c]ulturally and linguistically appropriate CAHPS survey reports can potentially help reduce the observed racial/ethnic differences in plan enrollment patterns." This study also warns that if white and English speakers are more likely to use the results of these surveys, it is possible that the results could reinforce the concentration of minorities in plans with lower assessments of care. Few studies examine the response bias within racial/ethnic groups for CAHPS surveys, but the one study that was cited seems to say that the bias is small. However, studies cite that the Spanish version of CAHPS is significantly less reliable and valid than the English versions. Cultural differences may influence response style in surveys and limit the ability to make comparisons between respondents of different racial/ethnic groups.
Efforts are being undertaken to increase the cultural competency of the CAHPS surveys. The Commonwealth Fund recently awarded money to the University of Florida, Principal Investigator Robert Weech-Maldonado, to develop and test the Patient Assessments of Cultural Competency Survey. "The CAHPS surveys have been used to assess racial, ethnic, and linguistic differences in patients' experiences with care. There are concerns, however, that the surveys do not fully capture the aspects of the care experience that are particularly relevant to minority patients, such as access to language services and perceived discrimination. The goal of this project is to test, validate, and disseminate a new survey – the Patient Assessments of Cultural Competency (PACC) – that addresses issues of cultural competency." The project team will create a short version of this survey to serve as a supplemental module for the CAHPS instruments. AHRQ and the National Committee for Quality Assurance intend to collaborate on the dissemination of the PACC survey. This survey should be completed in February 2009.
In addition to linguistic and cultural concerns, CAHPS surveys need to be readable and understandable for low-literacy groups. Currently, the CAHPS surveys range from a sixth to eighth grade reading level. If reading level is a problem, CAHPS survey administrators can conduct a survey over the phone. In order to increase readability, some studies are looking into creating an illustrated version of the CAHPS survey. However, some say that the telephone method may be easier and cheaper to administer than an illustrated CAHPS.
Is the CAHPS Survey Useful and Beneficial for Consumers of health care?
CAHPS is designed to be utilized by consumers as a tool for making health care decisions. However, consumers may not be aware that this information exists. In a 2005 Mathematica report, the authors state that the literature indicates that most consumers are unaware of the publicly available quality information. Efforts are being made to make consumers aware of CAHPS information, but current consumers still make decisions based primarily on information from family members, friends, or their physician and have a lower level of trust for published data.
CAHPS data is said to be easier for consumers to understand and relate to than report cards based on clinical measures. Studies report that consumers value information and demand the ability to compare health plan choices. Focus group evidence indicates that consumers want patient perspectives on care ratings for hospitals. However, while many consumers may report that this information is valuable, many studies conclude that the valued CAHPS results do not affect choices. Studies of Iowa and New Jersey Medicaid beneficiaries concluded that CAHPS did not affect health plan choices. The Mathematica literature review reports a mixed finding on the effects of CAHPS on actual consumer decision-making. Some studies show that more educated and informed consumers are more likely to use this information.
While the effect of CAHPS data on consumer actions is hard to see, the effects of this data on health providers is apparent. CAHPS literature suggests that the stakeholder role is becoming increasingly important. During the first stage of CAHPS, health plan providers became very interested in using CAHPS as a tool for quality improvement. Many studies report that health plans are using CAHPS to identify areas of needed improvement and as a rating mechanism to compare their plans with their competitors. The majority of the literature on CAHPS effectiveness focuses on how health plans are using the CAHPS survey data and how best to use the CAHPS survey data for quality improvement.
The underlying premise behind the usefulness of CAHPS for beneficiaries is that people will be better off if they have more information about their health choices. CAHPS seeks to provide a different set of information; not information about quality measures, but about a patient's perspective of global ratings, and experiences. In order to effectively capture accurate information from patients, CAHPS survey administrators need to be aware of the effects that proxies have on data, the issues with self-reported data, and ensure that the surveys are culturally competent. CAHPS data must address these concerns in order to be reliable for patients. Research shows that patients value and understand CAHPS ratings more than they understand complicated clinical measures. However, patients are not generally utilizing CAHPS data to help them make health care choices. Much research is being conducted on how to design an accurate survey tool, and many studies are being undertaken to examine how to increase the utility of CAHPS for health plans and health providers, however there seems to be less focus on creating a CAHPS report that is utilized by patients. The effectiveness of CAHPS data on patient choices has yet to be fully realized.
 Denise D. Quigley, Shelley H. Wiseman, and Donna O. Farley, Improving Performance for Health Plan Customer Service. RAND Health working paper, August 2007. p 15. available at http://www.rand.org/pubs/working_papers/WR517/.
 Charles Darby, Christine Crofton, and Carolyn Clancy , “Consumer Assessment of Health Providers and Systems (CAHPS): Evolving to Meet Stakeholder Needs.” American Journal of Medical Quality 2006; 21; 144.
 Implementation of Medicare CAHPS Fee-for-Service Survey: Final Report for the 2003 Survey. Prepared by RTI International and RAND, March 2005. available at http://www.cms.gov/CAHPS/Downloads/03_2003FFSCAHPSDataCollectionMethodsRPT.pdf
 Implementation of Medicare CAHPS Fee-for-Service Survey, supra note 6. “Proxy respondents rely on observable factors, such as counts or the presence or absence of a symptom, suggesting smaller discrepancies between self-and proxy-reports for objective or observable measures than for subjective measures.”
 Marc N. Elliott, Megan K. Beckett, Kelly Chong, Katrin Hambarsoomians, and Ron D. Hays, How do Proxy Responses and Proxy-Assisted Responses Differ from what Medicare Beneficiaries Might Have Reported about Their Health Care? Health Services Research 43:3 (June 2008). Available at http://findarticles.com/p/articles/mi_m4149/is_3_43/ai_n25467722.
 Implementation of Medicare CAHPS Fee-for-Service Survey, supra note 6
 How do Proxy Responses and Proxy-Assisted Responses Differ? Supra note 11 at 845
 A. Lynn Snow, et al. Proxies and Other External Raters: Methodological Considerations. Health Services Research 40:5, Part II (October 2005). page 1687. available at http://www3.interscience.wiley.com/journal/118685444/abstract?CRETRY=1&SRETRY=0.
 “Evolving to Meet Stakeholder Needs”, supra note 4 at 144.
https://www.cahps.ahrq.gov/content/cahpsOverview/faqanswer.asp?faq_id=89&showanswer=1&viewall=0¤t_cat_id=1&cat_id=1&keyword=&hassub=0#b_89 “One would not expect a large correlation between CAHPS surveys and clinical performance indicators because they provide complementary information about care. However, quality of care as measured in the CAHPS surveys should correlate significantly with aspects of clinical performance that are similar to the CAHPS domains. Indeed, some studies have found that patient reports about care are moderately related to HEDIS measures and predictive of clinical outcomes. Other studies demonstrate that some performance indicators are unrelated to CAHPS measures.”
 For a list of articles on this subject, see AHRQ’s Frequently Asked Questions, “Is there a link between performance on CAHPS surveys and clinical performance?” Id.
 “Race/Ethnicity, Language, and Patients’ Assessments of Care in Medicaid Managed Care”. Robert Weech-Maldonado, Leo S. Morales, Mar Elliot, Karen Spritzer, Grant Marshall, and Ron D. Hays. Health Services Research June 2003; 38:3, 799-808. and “Health Plan Effects on Patient Assessments of Medicaid Managed Care Among Racial/Ethnic Minorities” Robert Weech-Maldonado, Marc Elliot, Leo S. Morales, Karen Spritzer, Grant Marshall, Ron D. Hays. J Gen Intern Med 2004; 19: 136-145.
*Note, most of the studies on racial/ethnic bias have been done by Morales and Hays.
 “Health Plan Effects on Patient Assessments of Medicaid Managed Care Among Racial/Ethnic Minorities” supra note 19.
 “Variation in Racial and Ethnic Differences in Consumer Assessments of Health Care”, Nicole Lurie, Chunliu Zhan, Judith Sangl, Arlene S. Bierman, and Edward S. Sekscenski. Am J Manag Care. 2003; 9:502-509. The study cited is Morales LS, Reise SP, Hays RD “Evaluating the equivalence of health care ratings by whites and Hispanics”. Med Care. 2000; 38: 517-527.
 Tim Lake, Chris Kvam, and Marsha Gold. “Literature Review: Using Quality Information for Health Care Decisions and Quality Improvement, Final Report”. Mathematica Policy Research. May 2005.
 Id. at xii.
 Mark Spranca, David E. Kanouse, Marc Elliott, Pamela Farley Short, Donna O. Farley and Ron D. Hays, “Do Consumer Reports of Health Plan Quality Affect Health Plan Selection?” Health Services Research 35:5 Part I (December 2000).
 Harman Jordan, Alan White, Catherine Joseph, and Darcy Carr. “Costs and Benefits of HCAHPS”, Final Report. Abt Associates Inc. prepared for CMS October 5, 2005. page 9
 Leo S. Morales et al. “Applicability of CAHPS to PPO organizations in the US: Discussion of industry concerns.” International Journal for Quality in Health Care 2004; Volume 16, Number 3 pp. 219-227. available at http://intqhc.oxfordjournals.org/cgi/reprint/16/3/219.
 “Using Quality Information for Health Care Decisions and Quality Improvement”, supra note 26 at xii.
 Denise Quigley, et. al. “The Utility of CAHPS for Health Plans.” Rand Health working paper series. December 2003. p. 9