Controlling for race/ethnicity: a comparison of California commercial health plans CAHPS scores to NCBD benchmarks
© Zweifler et al; licensee BioMed Central Ltd. 2010
Received: 25 February 2009
Accepted: 25 January 2010
Published: 25 January 2010
Because California has higher managed care penetration and the race/ethnicity of Californians differs from the rest of the United States, we tested the hypothesis that California's lower health plan Consumer Assessment of Healthcare Providers and Systems (CAHPS®) survey results are attributable to the state's racial/ethnic composition.
California CAHPS survey responses for commercial health plans were compared to national responses for five selected measures: three global ratings of doctor, health plan and health care, and two composite scores regarding doctor communication and staff courtesy, respect, and helpfulness. We used the 2005 National CAHPS 3.0 Benchmarking Database to assess patient experiences of care. Multiple stepwise logistic regression was used to see if patient experience ratings based on CAHPS responses in California commercial health plans differed from all other states combined.
CAHPS patient experience responses in California were not significantly different than the rest of the nation after adjusting for age, general health rating, individual health plan, education, time in health plan, race/ethnicity, and gender. Both California and national patient experience scores varied by race/ethnicity. In both California and the rest of the nation Blacks tended to be more satisfied, while Asians were less satisfied.
California commercial health plan enrollees rate their experiences of care similarly to enrollees in the rest of the nation when seven different variables including race/ethnicity are considered. These findings support accounting for more than just age, gender and general health rating before comparing health plans from one state to another. Reporting on race/ethnicity disparities in member experiences of care could raise awareness and increase accountability for reducing these racial and ethnic disparities.
In 1995 the United States Agency for Healthcare Research and Quality launched what is now the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) program in an effort to standardize health plan evaluation. The CAHPS Health Plan Survey, a voluntary survey designed to capture data on member experiences with care as a measure of health plan quality, is one of a family of CAHPS surveys designed for specific health care settings . The CAHPS Health Plan Survey is incorporated into the National Committee for Quality Assurance (NCQA) accreditation process for health plans . Consumer assessments are playing an increasing role as quality indicators and in enrollee guides . U.S. News & World reports have ranked California commercial health plans lower than national counterparts in rankings based on performance on NCQA clinical quality measures for prevention and treatment, as well as CAHPS patient experience responses that are adjusted for variables including age, general health rating, and gender [4, 5]. Only one California health plan was ranked in the top 25 percent of commercial health plans in their 2008 report. Even among California's highest ranked health plans, consumer assessment scores are consistently lower than prevention and treatment scores .
In 2007, California was home to 12 percent of the American population . However, it accounts for one quarter of the total U.S. health-maintenance organization (HMO) enrollment . HMOs can be broadly defined as health plans that offer prepaid, comprehensive health coverage for both hospital and physician services. Close to half of California's 35 million population is enrolled in an HMO . Since higher HMO penetration in California may forecast higher HMO penetration in the rest of the nation, it is important to understand why member scores are lower in California.
One plausible explanation for lower HMO member ratings of care in California is that the racial and ethnic composition of California differs from the rest of the nation. It may be appropriate to consider race/ethnicity in member surveys because the Institute of Medicine includes "equity" as one of the six domains of health care quality . CAHPS survey results are case-mix adjusted for age, gender, and self-reported health status, but although CAHPS surveys collect data on race/ethnicity, no adjustment is made for race/ethnicity . Other research suggests individual health plans have a greater influence on member experiences of care than race/ethnicity, though this too is not a CAHPS-specific recommended adjustment [10, 11].
Racial and ethnic disparities in survey responses are well documented. A 2003 study looking at Medicaid CAHPS data determined that consumer assessments of managed care varied by race, ethnicity and language, with racial/ethnic minorities reporting lower quality of care . A study using the Medical Expenditure Panel Survey to assess enrollees in Medicaid managed care programs found Hispanics and Asian Americans reported less satisfaction with care than Whites . Findings from the Commonwealth Fund 2001 Health Care Quality Survey showed only 56 percent of Hispanics and 45 percent of Asian Americans reported being "very satisfied" with their health care versus 65 percent of Whites . Additionally, 15 percent of African Americans, 13 percent of Hispanics and 11 percent of Asian Americans felt they would have received better care if they were a different race or ethnicity.
A number of studies have shown that Asians and Pacific Islanders report different experiences with care than other races/ethnicities. Weech-Maldonado et al. reported Asian non-English speakers presented the lowest ratings of care of all the ethnicities and "had more negative reports and ratings of care" than Asian English speakers, whose ratings were similar to Whites . Phillips et al. found Asian Americans the ethnic group most likely to report dissatisfaction with the quality of care provided by their usual source of care . Taira et al. reported Asian Americans rated overall satisfaction, and 10 of 11 scales assessing primary care significantly lower than Whites . They suggested that controlling for Asian-American ethnicity "could significantly improve the profiles of physicians with a large percentage of Asian-American patients." In a managed care setting, Asian patients as a group have been found to be "significantly less satisfied" with various aspects of primary care performance than other ethnicities .
California's diverse demographics may significantly impact its consumer assessment scores. Notably, there are more non-Whites than Whites in California with 35.9 percent Hispanic/Latino, 12.4 percent Asian, and 6.7 percent Black . In 2007, California ranked 2nd in the nation for percent of the total population who are Asian alone .
This study will test the hypothesis that California commercial health plan CAHPS survey results are attributable to the racial and ethnic composition of the state's population. California commercial health plan CAHPS survey responses will be compared to the 2005 National CAHPS Benchmark Database (NCBD) for selected measures. CAHPS survey results stratified by race/ethnicity will be presented and the impact of controlling for race/ethnicity as case-mix adjustments for California and nationally, will be discussed.
Consumer assessments of healthcare providers and systems (CAHPS®) database questions and scales used
Global Rating Question
What number would you use to rate your personal doctor or nurse?
What number would you use to rate your health plan?
What number would you use to rate all your health care in the last 12 months?
Composite: Staff Courtesy and Respect
In the last 12 months, how often did office staff at a doctor's office or clinic treat you with courtesy and respect?
In the last 12 months, how often were office staff at a doctor's office or clinic as helpful as you thought they should be?
Composite: Doctor Communication
In the last 12 months, how often did doctors or other health providers listen carefully to you?
In the last 12 months, how often did doctors or other health providers explain things in a way you could understand?
In the last 12 months, how often did doctors or other health providers show respect for what you had to say?
In the last 12 months, how often did doctors or their health providers spend enough time with you?
We used the provided CAHPS categories for all of our demographic variables except race and insurance plan type. We combined the race and Hispanic variables in the database to incorporate Hispanic ethnicity into our race variable. Anyone who reported they were Hispanic in the Hispanic variable was coded as such regardless of race. Anyone reporting non-Hispanic in the Hispanic variable was coded as the race they specified in the race variable so our race categories were mutually exclusive. The type of insurance variable was collapsed into two categories: HMO and non-HMO. The non-HMO includes the CAHPS categories of POS, PPO, combined and other. Using the state of residence we created a variable to distinguish people in California from those in the rest of the nation (non-California).
In order to facilitate reporting and analysis, we chose to collapse each global rating score into two categories: highly rated or not highly rated. We followed the recommendation of Weech-Maldonado et al. (2008) and anyone responding either 9 or 10 on the worst to best scale was categorized as highly rated on that measure while all lower scores were categorized as not highly rated . We followed CAHPS directions to create our composite variables . We chose to equally weight the questions in each composite and added together all scores to get a possible total score of 16 for doctor communication (composite consists of four items) and 8 for staff courtesy, respect and helpfulness (composite consists of two items). We then collapsed these into highly rated categories based on scoring at least 90% of all possible points. Respondents scoring 14 or more on the doctor communication composite were categorized as highly rated. Those scoring 7 or 8 on the staff composite were categorized as highly rated. Scores lower than these were categorized as not highly rated.
We compared California to the rest of the nation using chi-square tests for the categorical demographic data. Chi-square tests were also used to look for differences in highly rated results for each global rating and both composite reports of care by race categories. We also compared California to the rest of the nation on each of these five measures using chi-square. Stepwise multiple logistic regression was used to see if ratings in California differed from all the other states combined (rest of the nation) on the five measures. Five regressions were done using the dependent measures of doctor rating, plan rating and care rating as well as reports on doctors communication and staff courtesy, respect and helpfulness. We used the following independent variables in each regression: age group (3 categories), education (6 categories), gender (2 categories), general health rating (5 categories), individual health plan (254 categories), insurance type (HMO or non-HMO), location (California or non-California), race/ethnicity (5 categories), and time in plan (4 categories). Fisher's exact tests were used to compare overall rankings of individual health plans for California to the rest of the nation using the logistic regression odds ratio results. Categories created based on the results of comparison to a New York HMO were: statistically higher, not statistically different, and statistically lower.
Respondent characteristics as defined and categorized by CAHPS®
California n = 9,952
Rest of nation* n = 113,320
18 - 34
35 - 64
Insurance type (%)
Time in plan (%)
< 1 year
1 < 2 years
2 < 5 years
≥ 5 years
Education level (%)
≤ 8th grade
Some high school
High school graduate
Some college (AA)
College graduate (BA/BS)
More than 4 year degree
General Health Rating (%)
Do you have one person you think of as your personal doctor or nurse?
In the last 12 months (not counting times you went to an emergency room), how many times did you go to a doctor's office or clinic to get care for yourself?
Percentage highly rated responses to selected CAHPS® global ratings and composites for California and the nation
Odds ratio (95% confidence interval) results for 5 logistic regression models
Model dependent variable - Global Ratings
18 - 34 years
35 - 64 yearsR
≥ 65 years
General Health Rating
< 8th grade
Some high school
High school graduateR
BA/BS college degree
Individual health plan
254 plans not included due to size constraints
Time in plan
< 1 year
1 to < 2 years
2 to < 5 years
≥ 5 yearsR
Model dependent variable - Composites
18 - 34 years
35 - 64 yearsR
≥ 65 years
General Health Rating
< 8th grade
Some high school
High school graduateR
BA/BS college degree
Individual health plan
254 plans not included due to size constraints
Time in plan
< 1 year
1 to < 2 years
2 to < 5 years
≥ 5 yearsR
We hypothesized that racial and ethnic differences in member experience of care scores, coupled with the higher concentration of Hispanics and Asian/Pacific Islanders in California would explain the lower scores California commercial health plans have received on CAHPS surveys. However, our analysis of adult enrollees in commercial health plans found that race/ethnicity was only one of multiple variables that influenced responses to selected CAHPS questions in California and the rest of the nation. When these variables were all accounted for, CAHPS scores for California health plans were no different than the rest of the nation. We found that the variables that contributed first in our stepwise logistic regression were age, general health rating, education, and health plan. Time in plan, race/ethnicity, and gender were also significant, but they were consistently less influential than the four variables noted above. Our findings suggest that national comparisons of patient experiences of care should adjust for a broader range of variables.
Californians enrolled in commercial health plans
Although we found that California commercial health plan enrollee responses to selected questions in the CAHPS survey did not differ from the rest of the nation, California health plan enrollees did differ in several respects. First, there is a higher percentage of Californians enrolled in HMOs. Consequently, it is possible that Californians are more familiar with managed care and/or may have different expectations of HMO care than the rest of the nation. This familiarity and/or differing expectations could be reflected in CAHPS responses. Testing this hypothesis would require linking HMO penetration data not available in the CAHPS database, with CAHPS responses stratified by geography. It is possible that patient experience responses from commercial HMO enrollees differ from how enrollees in non-participating HMOs would respond. However, patient experience CAHPS surveys are mandated if an HMO seeks NCQA accreditation, and over 90% of all commercial HMO enrollees in California are in plans that have received NCQA accreditation .
California also differs from the rest of the nation in that a higher percentage of patients are cared for by large medical groups that contract with health plans to provide the majority of managed care services . This "delegated model" transfers responsibility for many patient care decisions including authorization of diagnostic, inpatient, and specialty services from the health plan to the medical group. This may shield health plans from patient frustrations associated with accessing health care services while exposing the physicians that constitute the medical groups to more critical scrutiny and hence lower CAHPS scores. In unadjusted scores, Californians rated their health plan itself higher than enrollees around the nation rated their health plan. Conversely, California health plan enrollees rated their doctor, and their care, lower than health plan enrollees in the rest of the nation. However, these differences were not significant after controlling for other variables in our logistic regression. Furthermore, if lower patient reports of care were attributable to the delegated model, we would expect to see differences in CAHPS responses between Kaiser's group model and the other California commercial health plans that rely on the delegated model. In contrast, CAHPS scores for Kaiser, a group model HMO that serves close to one third of the commercial HMO members in California, are similar to CAHPS scores for other California commercial health plans [24, 25].
We found that California commercial health plan enrollees rated their experiences of care similarly to commercial enrollees in the rest of the nation, but that may not mean the care itself is comparable. However, on a measure of clinical quality, the Health Plan Employer Data Information Set (HEDIS®), California commercial health plans scores are comparable to health plans across the nation . Commercial enrollees in California differed from the rest of the nation in that Californians were less likely to have a personal doctor or to have visited a doctor's office or clinic in the last 12 months. This could suggest that Californians are healthier, have less access to care, or are otherwise incentivized to not have a personal doctor or seek care. It is unclear how these differences would impact our findings.
Our comparison of California to the rest of the nation's commercial CAHPS scores showed that race/ethnicity was only part of the reason scores differ. We found consistent variation in CAHPS scores between racial and ethnic groups. In both California and the rest of the nation, Blacks were significantly more likely to rate their doctor, their plan, and their care higher than Whites; Asians were more likely to rate their care, courtesy, understanding, and respect lower than Whites; and Hispanics were more likely to rate their plan higher than Whites.
One explanation for why racial and ethnic groups score their doctor, care, courtesy, understanding, and respect lower in California, as well as the nation as a whole, is discordance between physicians and patients. While California's population is very diverse, its physicians do not reflect the state's racial and ethnic distribution . It is possible that CAHPS scores that assess interactions with their physician are lower if the physician is from a different racial or ethnic group. Some but not all studies suggest that patient-physician concordance impact patient ratings of care [28–30]. However, we are unable to investigate patient-physician concordance in the CAHPS findings because physician race/ethnicity is not collected.
It is also possible that the race/ethnicity of commercial health plan CAHPS respondents does not reflect the demographics of all commercial enrollees because certain racial/ethnic groups may be less comfortable responding to surveys than other groups. We are unable to account for this potential source of variation because we do not have racial or ethnic data for commercial health plan enrollees who do not respond to the CAHPS survey. We also cannot comment on whether the differences in CAHPS scores reflect the perceptions of all Californians in the different racial/ethnic groups, or only those Californians who are enrolled in commercial health plans.
Our analysis of commercial CAHPS scores does not explain why responses vary by race/ethnicity. If CAHPS responses vary by race/ethnicity because of differences in how CAHPS questions are interpreted rather than the care received, then not controlling for race/ethnicity would confound comparing member ratings of experiences of care in geographic regions with differing concentrations of Asians, Blacks and Hispanics. However, if health care does truly vary by race/ethnicity, then controlling for race/ethnicity would obscure real differences in care that would be important to track. It is of interest to note that another large scale member satisfaction survey in California, the Patient Assessment Survey sponsored by The California Cooperative Healthcare Reporting Initiative does control for race/ethnicity .
Our analysis has several limitations. Member participation in CAHPS surveys is voluntary, so participants may vary from all health plan enrollees in demographics and in perceptions of care. Most health plans are confined to individual states, limiting our ability to compare health plans across state lines . CAHPS surveys already control for age, general health rating, and gender, but our analysis found that other variables including individual health plan, education, time in plan as well as race/ethnicity influenced responses to selected CAHPS questions. There may be other variables influencing CAHPS responses that we did not consider. For example, intrastate geographic differences in access to care and participation rates in CAHPS may account for some of the variation in responses.
Our analysis was limited to adult commercial enrollees. We did not look at health plan enrollees with other types of health insurance such as Medicare, Medicaid, or SCHIP, or at other age groups such as children. However, comparing experiences of care ratings across product lines would be difficult to interpret because of differences in reimbursement formulas and in enrollees.
If race/ethnicity information was routinely collected by commercial health plans we could determine if CAHPS respondents differed from commercial health plan enrollees as a whole. Race/ethnicity data for commercial health plan enrollees would also allow us to look for disparities in other types of quality metrics such as HEDIS®. Regardless of whether the race/ethnicity differences in member scores are driven by cultural biases in survey responses or are due to real differences in care, our analysis supports the wider application of comparisons that consider race/ethnicity.
California health plan enrollees rate their care similarly to health plan enrollees in the rest of the nation when we controlled for age, general health rating, education, individual plan, race/ethnicity, time in plan, and gender characteristics. Standardizing populations for these variables prior to comparing individual health plans or health plans by state would lessen the likelihood of ascribing differences in member reports of experiences of care to the health plan or state, when they may have more to do with the enrollees themselves.
Stratifying member experiences of care by race/ethnicity highlighted important differences in CAHPS ratings between different racial and ethnic groups in both California and the nation. Reporting on race/ethnicity disparities in member experiences of care could raise awareness and increase accountability for reducing racial and ethnic disparities, especially among individual health plans. Wider dissemination of public reporting which accounts for race/ethnicity and incentive programs that reward reducing disparities in care may help achieve these objectives.
Consumer Assessments of Healthcare Providers and Systems (CAHPS®) is a registered trademark of the U.S. Agency for Healthcare Research and Quality. We thank the National CAHPS® Benchmarking Database and Dale Shaller for making this data available.
- CAHPS Overview. [https://www.cahps.ahrq.gov/content/cahpsOverview/Over_Program.asp?p=101&s=12]
- Specifications for Survey Measures. 3: [http://web.ncqa.org/tabid/437/Default.aspx]
- 2007 Healthcare Quality Report Card. [http://www.opa.ca.gov/report_card/]
- America's Best Health Plans. [http://health.usnews.com/health/health-plans]
- U.S. Department of Health and Human Services, Agency for Healthcare Research and Quality: CAHPS Health Plan Survey and Reporting Kit 2007. Instructions for Analyzing CAHPS Data: Using the CAHPS Analysis Program Version 3.6. 2007Google Scholar
- U.S. Census Bureau: Annual Estimates of the Population for the United States, Regions, States, and Puerto Rico: April 1, 2000 to July 1, 2007. 2007Google Scholar
- California: Total HMO Enrollment. 2007, [http://www.statehealthfacts.org/profileind.jsp?ind=348&cat=7&rgn=6]
- California: State HMO Penetration Rate. 2007, [http://www.statehealthfacts.org/profileind.jsp?ind=349&cat=7&rgn=6]
- The Committee on Quality of Health Care in America, Institute of Medicine: Crossing the Quality Chasm: A New Health System for the 21st Century. 2001, Washington, D.C.: The National Academies PressGoogle Scholar
- Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ: Relationship between quality of care and racial disparities in Medicare health plans. JAMA. 2006, 296: 1998-2004. 10.1001/jama.296.16.1998.View ArticlePubMedGoogle Scholar
- Weech-Maldonado R, Elliott MN, Morales LS, Spritzer K, Marshall GN, Hays RD: Health plan effects on patient assessments of Medicaid managed care among racial/ethnic minorities. J Gen Intern Med. 2004, 19: 136-145. 10.1111/j.1525-1497.2004.30235.x.PubMed CentralView ArticlePubMedGoogle Scholar
- Phillips KA, Mayer ML, Aday LA: Barriers to care among racial/ethnic groups under managed care. Health Aff (Millwood). 2000, 19: 65-75. 10.1377/hlthaff.19.4.65.View ArticleGoogle Scholar
- Collins KS, Hughes DL, Doty MM, Ives BL, Edwards JN, Tenney K: Diverse Communities, Common Concerns: Assessing Health Care Quality for Minority Americans. 2002, The Commonwealth FundGoogle Scholar
- Weech-Maldonado R, Morales LS, Elliott M, Spritzer K, Marshall G, Hays RD: Race/ethnicity, language, and patients' assessments of care in Medicaid managed care. Health Serv Res. 2003, 38: 789-808. 10.1111/1475-6773.00147.PubMed CentralView ArticlePubMedGoogle Scholar
- Taira DA, Safran DG, Seto TB, Rogers WH, Kosinski M, Ware JE, Lieberman N, Tarlov AR: Asian-American patient ratings of physician primary care performance. J Gen Intern Med. 1997, 12: 237-242. 10.1007/s11606-006-5046-0.PubMed CentralView ArticlePubMedGoogle Scholar
- Murray-Garcia JL, Selby JV, Schmittdiel J, Grumbach K, Quesenberry CP: Racial and ethnic differences in a patient survey: patients' values, ratings, and reports regarding physician primary care performance in a large health maintenance organization. Med Care. 2000, 38: 300-310. 10.1097/00005650-200003000-00007.View ArticlePubMedGoogle Scholar
- U.S. Census Bureau: State and County QuickFacts: California. 2008Google Scholar
- U.S. Census Bureau, American FactFinder: United States and States, Percent of the Total Population Who Are Asian Alone: 2007. 2007Google Scholar
- Patient Assessment Survey 2010 Project Information: Registration Package. [http://www.cchri.org/programs/PAS_2010/2010_PAS_Invitation_Package_Hardcopy_FINAL.pdf]
- Weech-Maldonado R, Elliott MN, Oluwole A, Schiller KC, Hays RD: Survey response style and differential use of CAHPS rating scales by Hispanics. Med Care. 2008, 46: 963-968. 10.1097/MLR.0b013e3181791924.PubMed CentralView ArticlePubMedGoogle Scholar
- National CAHPS Benchmarking Database: 2005 CAHPS Health Plan Survey Chartbook. 2005, Rockville, MD: Agency for Healthcare Research and QualityGoogle Scholar
- About the HMO Ratings. [http://opa.ca.gov/report_card/hmoabout.aspx]
- The California Association of Physician Groups. [http://www.capg.org/home/index.asp]
- The 2006 California Statewide HMO & Special Programs Enrollment Study. [http://www.cattaneostroud.com/2006HMO.htm]
- HMO Ratings At-a-Glance. [http://www.opa.ca.gov/report_card/hmorating.aspx]
- Healthcare Quality Report Card, California HMO/POS plans vs the National Average. 2006, [http://www.opa.ca.gov/rc2006/averages.aspx]
- Dower C, McRee T, Grumbach K, Briggance B, Mutha S, Coffman J, Vranizan K, Bindman A, O'Neil E: The Practice of Medicine in California: A Profile of the Physician Workforce. 2001, San Francisco CA: California Workforce Initiative at the UCSF Center for the Health ProfessionsGoogle Scholar
- Blanchard J, Nayar S, Lurie N: Patient-provider and patient-staff racial concordance and perceptions of mistreatment in the health care setting. J Gen Intern Med. 2007, 22: 1184-1189. 10.1007/s11606-007-0210-8.PubMed CentralView ArticlePubMedGoogle Scholar
- Saha S, Komaromy M, Koepsell TD, Bindman AB: Patient-physician racial concordance and the perceived quality and use of health care. Arch Intern Med. 1999, 159: 997-1004. 10.1001/archinte.159.9.997.View ArticlePubMedGoogle Scholar
- Stevens GD, Shi L, Cooper LA: Patient-provider racial and ethnic concordance and parent reports of the primary care experiences of children. Ann Fam Med. 2003, 1: 105-112. 10.1370/afm.27.PubMed CentralView ArticlePubMedGoogle Scholar
- Patient Assessment Survey (PAS). [http://www.cchri.org/programs/programs_pas.html]
- HMO Members by Type of Insurance. [http://www.opa.ca.gov/report_card/stat.aspx?Chart=MembersByInsurance]
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.